Keep diagnostics release to one PDF report
This commit is contained in:
aseimel
@@ -8,4 +8,4 @@ Rscript diagnostics/generate_diagnostics.R
|
||||
|
||||
The command requires completed model/post-estimation outputs. If those outputs are outside the repository root, set `PARTY2D_OUTPUTS_DIR` before running the script.
|
||||
|
||||
The main reports are `diagnostics_report.md` and `diagnostics_report.pdf`; both are copied into `data/releases/` for the release files.
|
||||
The report file is `diagnostics_report.pdf`; the same PDF is copied into `data/releases/` for the release files.
|
||||
|
||||
@@ -1,241 +0,0 @@
|
||||
# Diagnostics report
|
||||
|
||||
Generated: 2026-06-15 16:40:02 CEST
|
||||
Release: v0
|
||||
Model positions source: `party_positions_2026-06-12_16-35-01.csv`
|
||||
|
||||
## Purpose
|
||||
|
||||
The purpose of this report is to provide the appendix-style diagnostics that document how the released party-position estimates are constructed, checked, and validated. The main article reports the central validation evidence; this report keeps the larger technical tables with the release so readers can inspect item coverage, source coverage, coding orientation, harmonization, convergence, posterior uncertainty, and validation details in one reproducible place.
|
||||
|
||||
## Overview
|
||||
|
||||
This report follows the structure of the technical appendix material: data and item coverage, coding and scale orientation, party-union harmonization, construct checks, model convergence, and validation. It is generated from the model-ready inputs and completed model outputs; it is not part of the raw-data setup workflow.
|
||||
|
||||
## Data and item coverage
|
||||
|
||||
The model combines text-coded item counts, dimension-specific expert placements, and general left-right expert placements. Text items enter as positive/sample counts, expert items enter as aggregated ratings with scale and expert-count information, and general left-right ratings anchor the relationship between the two dimensions.
|
||||
|
||||
| source_file | dimension | items |
|
||||
| --- | --- | --- |
|
||||
| expert.csv | cultural cosmopolitan--traditionalist | 7 |
|
||||
| expert.csv | economic left-right | 5 |
|
||||
| lr_data.csv | general left-right | 3 |
|
||||
| text_data.csv | cultural cosmopolitan--traditionalist | 11 |
|
||||
| text_data.csv | economic left-right | 6 |
|
||||
|
||||
### Source coverage
|
||||
|
||||
| source_file | source | items | observations | party_years | parties | countries | min_year | max_year |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| expert.csv | CHES | 2 | 2639 | 1320 | 390 | 44 | 1999 | 2024 |
|
||||
| expert.csv | GPS | 2 | 538 | 271 | 271 | 62 | 2019 | 2019 |
|
||||
| expert.csv | POPPA | 1 | 413 | 413 | 225 | 31 | 2018 | 2023 |
|
||||
| expert.csv | V-Party | 7 | 21488 | 3076 | 589 | 65 | 1970 | 2019 |
|
||||
| lr_data.csv | CHES | 1 | 1320 | 1320 | 390 | 44 | 1999 | 2024 |
|
||||
| lr_data.csv | Morgan | 1 | 471 | 471 | 72 | 11 | 1945 | 1973 |
|
||||
| lr_data.csv | POPPA | 1 | 416 | 416 | 225 | 31 | 2018 | 2023 |
|
||||
| text_data.csv | Manifesto Project | 8 | 36008 | 4501 | 713 | 65 | 1920 | 2025 |
|
||||
| text_data.csv | PolDem | 9 | 2194 | 406 | 93 | 15 | 1972 | 2017 |
|
||||
|
||||
## Data coding and item orientation
|
||||
|
||||
All indicators are oriented toward the two reported dimensions: economic left-right and cultural cosmopolitan--traditionalist. For interpretability, generated diagnostics report whether higher observed values point toward the public high pole or are reversed for reporting. Original source variable names are preserved in the tables.
|
||||
|
||||
### Reversed items
|
||||
|
||||
| item | source | dimension | higher_values_indicate | observations | min_year | max_year |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| internationalism_manifesto | Manifesto Project | cultural cosmopolitan--traditionalist | cosmopolitan | 4501 | 1920 | 2025 |
|
||||
| multiculturalism_manifesto | Manifesto Project | cultural cosmopolitan--traditionalist | cosmopolitan | 4501 | 1920 | 2025 |
|
||||
| economic_intervention_manifesto | Manifesto Project | economic left-right | pro_welfare | 4501 | 1920 | 2025 |
|
||||
| social_services_manifesto | Manifesto Project | economic left-right | pro_welfare | 4501 | 1920 | 2025 |
|
||||
| cultlib_poldem | PolDem | cultural cosmopolitan--traditionalist | cosmopolitan | 299 | 1972 | 2017 |
|
||||
| euro_poldem | PolDem | cultural cosmopolitan--traditionalist | cosmopolitan | 93 | 1978 | 2017 |
|
||||
| europe_poldem | PolDem | cultural cosmopolitan--traditionalist | cosmopolitan | 217 | 1972 | 2017 |
|
||||
| immig_poldem | PolDem | cultural cosmopolitan--traditionalist | cosmopolitan | 236 | 1972 | 2017 |
|
||||
| welfare_poldem | PolDem | economic left-right | pro_welfare | 349 | 1972 | 2017 |
|
||||
|
||||
## Party unions and electoral coalitions
|
||||
|
||||
Alliance and union labels are handled through constituent mappings so the released party identifiers represent individual parties. Shared text evidence can inform constituent parties through the union mapping while expert data continue to constrain individual parties directly.
|
||||
|
||||
| metric | category | value |
|
||||
| --- | --- | --- |
|
||||
| constituent_mappings | all | 106 |
|
||||
| unique_union_or_alliance_ids | all | 76 |
|
||||
| unique_constituent_party_ids | all | 100 |
|
||||
| mappings_by_country | AR | 5 |
|
||||
| mappings_by_country | AU | 2 |
|
||||
| mappings_by_country | BA | 1 |
|
||||
| mappings_by_country | BE | 4 |
|
||||
| mappings_by_country | BG | 5 |
|
||||
| mappings_by_country | BR | 6 |
|
||||
| mappings_by_country | CH | 1 |
|
||||
| mappings_by_country | CL | 8 |
|
||||
| mappings_by_country | CZ | 4 |
|
||||
| mappings_by_country | DE | 3 |
|
||||
| mappings_by_country | EC | 1 |
|
||||
| mappings_by_country | EE | 3 |
|
||||
| mappings_by_country | ES | 3 |
|
||||
| mappings_by_country | FR | 4 |
|
||||
| mappings_by_country | GR | 3 |
|
||||
| mappings_by_country | HR | 4 |
|
||||
| mappings_by_country | IE | 1 |
|
||||
| mappings_by_country | IL | 2 |
|
||||
| mappings_by_country | IT | 8 |
|
||||
| mappings_by_country | LK | 3 |
|
||||
| mappings_by_country | LT | 4 |
|
||||
| mappings_by_country | LU | 1 |
|
||||
| mappings_by_country | LV | 6 |
|
||||
| mappings_by_country | MD | 1 |
|
||||
| mappings_by_country | ME | 1 |
|
||||
| mappings_by_country | MX | 2 |
|
||||
| mappings_by_country | NL | 1 |
|
||||
|
||||
## Party-family coverage
|
||||
|
||||
Party-family classifications are used for construct-validity diagnostics and coverage summaries. The table below reports coverage in the completed model output by family code.
|
||||
|
||||
| family | parties | party_years | countries | min_year | max_year |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| soc | 86 | 2892 | 37 | 1944 | 2025 |
|
||||
| con | 82 | 2425 | 34 | 1944 | 2024 |
|
||||
| lib | 81 | 2063 | 35 | 1944 | 2025 |
|
||||
| chr | 41 | 1596 | 24 | 1945 | 2025 |
|
||||
| com | 49 | 1241 | 27 | 1944 | 2025 |
|
||||
| right | 50 | 975 | 26 | 1946 | 2025 |
|
||||
| eco | 30 | 723 | 24 | 1960 | 2024 |
|
||||
| agr | 12 | 592 | 10 | 1944 | 2024 |
|
||||
| spec | 25 | 520 | 14 | 1949 | 2024 |
|
||||
| other | 2 | 40 | 2 | 1992 | 2024 |
|
||||
|
||||
### Construct-validity family means
|
||||
|
||||
Substantive party-family means provide a construct-validity check: families should follow the expected ordering on the economic left-right and cultural cosmopolitan--traditionalist dimensions.
|
||||
|
||||
| family_name | n_parties | n_obs | mean_economic | sd_economic | mean_cultural | sd_cultural |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| Communist/Far Left | 49 | 1241 | 0.12 | 0.09 | 0.383 | 0.18 |
|
||||
| Green/Ecological | 30 | 723 | 0.267 | 0.115 | 0.251 | 0.092 |
|
||||
| Social Democratic | 86 | 2892 | 0.331 | 0.124 | 0.378 | 0.138 |
|
||||
| Christian Democratic | 41 | 1596 | 0.596 | 0.119 | 0.529 | 0.14 |
|
||||
| Radical Right | 50 | 975 | 0.622 | 0.19 | 0.72 | 0.151 |
|
||||
| Conservative | 82 | 2425 | 0.652 | 0.178 | 0.538 | 0.143 |
|
||||
| Liberal | 81 | 2063 | 0.657 | 0.152 | 0.371 | 0.132 |
|
||||
|
||||
### Temporal-stability flags
|
||||
|
||||
The model permits movement through random walks, but unusually large one-year changes are flagged for inspection rather than treated as automatic errors.
|
||||
|
||||
| party_id | country | dimension | year_from | year_to | val_from | val_to | annual_change |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| 556 | LT | cultural cosmopolitan--traditionalist | 2019 | 2020 | 0.771 | 0.529 | 0.242 |
|
||||
| 1663 | IL | cultural cosmopolitan--traditionalist | 2021 | 2022 | 0.142 | 0.356 | 0.214 |
|
||||
| 455 | IL | cultural cosmopolitan--traditionalist | 1997 | 1998 | 0.457 | 0.643 | 0.186 |
|
||||
| 455 | IL | cultural cosmopolitan--traditionalist | 1996 | 1997 | 0.277 | 0.457 | 0.18 |
|
||||
| 8393 | LV | cultural cosmopolitan--traditionalist | 2018 | 2019 | 0.439 | 0.263 | 0.176 |
|
||||
| 281 | BE | cultural cosmopolitan--traditionalist | 1977 | 1978 | 0.671 | 0.843 | 0.173 |
|
||||
| 556 | LT | cultural cosmopolitan--traditionalist | 2001 | 2002 | 0.319 | 0.491 | 0.172 |
|
||||
| 964 | IS | economic left-right | 2016 | 2017 | 0.691 | 0.521 | 0.17 |
|
||||
| 298 | NL | cultural cosmopolitan--traditionalist | 2018 | 2019 | 0.761 | 0.593 | 0.168 |
|
||||
| 901 | FI | economic left-right | 1992 | 1993 | 0.486 | 0.647 | 0.161 |
|
||||
| 467 | SI | cultural cosmopolitan--traditionalist | 2018 | 2019 | 0.564 | 0.404 | 0.16 |
|
||||
| 1221 | IT | economic left-right | 2007 | 2008 | 0.56 | 0.4 | 0.159 |
|
||||
| 901 | FI | economic left-right | 1991 | 1992 | 0.327 | 0.486 | 0.159 |
|
||||
| 455 | IL | cultural cosmopolitan--traditionalist | 1998 | 1999 | 0.643 | 0.797 | 0.154 |
|
||||
| 1221 | IT | economic left-right | 2006 | 2007 | 0.709 | 0.56 | 0.149 |
|
||||
| 2211 | UA | economic left-right | 2006 | 2007 | 0.417 | 0.562 | 0.145 |
|
||||
| 631 | CH | economic left-right | 2018 | 2019 | 0.613 | 0.757 | 0.144 |
|
||||
| 298 | NL | cultural cosmopolitan--traditionalist | 2019 | 2020 | 0.593 | 0.735 | 0.142 |
|
||||
| 556 | LT | cultural cosmopolitan--traditionalist | 2000 | 2001 | 0.181 | 0.319 | 0.138 |
|
||||
| 1417 | IL | cultural cosmopolitan--traditionalist | 1968 | 1969 | 0.498 | 0.635 | 0.137 |
|
||||
|
||||
## Model convergence diagnostics
|
||||
|
||||
Convergence is assessed using split R-hat and effective sample size over monitored parameters.
|
||||
|
||||
| metric | count | percentage |
|
||||
| --- | --- | --- |
|
||||
| R-hat < 1.01 | 35258 | 100 |
|
||||
| R-hat 1.01-1.05 | 0 | 0 |
|
||||
| R-hat > 1.05 | 0 | 0 |
|
||||
| ESS > 1000 | 34853 | 98.85 |
|
||||
| ESS 400-1000 | 405 | 1.15 |
|
||||
| ESS < 400 | 0 | 0 |
|
||||
|
||||
### Convergence by parameter group
|
||||
|
||||
| dimension | parameters | mean_rhat | max_rhat | min_ess_bulk | mean_ess_bulk |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| cultural cosmopolitan--traditionalist | 17585 | 1.001 | 1.006 | 810.509 | 6889.246 |
|
||||
| economic left-right | 17585 | 1 | 1.004 | 670.708 | 5573.456 |
|
||||
| lr_country_offset | 65 | 1.001 | 1.004 | 1078.198 | 5434.868 |
|
||||
| lr_decade_offset | 8 | 1 | 1.001 | 2069.676 | 3372.016 |
|
||||
| lr_sigma | 3 | 1.001 | 1.002 | 1956.611 | 2582.401 |
|
||||
| lr_source_offset | 3 | 1 | 1 | 3477.519 | 3814.374 |
|
||||
| lr_weight | 3 | 1.002 | 1.002 | 1540.806 | 1560.217 |
|
||||
| mean_sigma | 6 | 1.005 | 1.009 | 534.811 | 918.502 |
|
||||
|
||||
## Posterior uncertainty
|
||||
|
||||
The completed party-position output reports posterior standard errors and interval endpoints for both dimensions. These summaries describe the typical uncertainty in the release file used by the report.
|
||||
|
||||
| rows | parties | countries | min_year | max_year | mean_economic_se | median_economic_se | mean_cultural_se | median_cultural_se |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| 17585 | 708 | 65 | 1944 | 2025 | 0.065 | 0.063 | 0.058 | 0.055 |
|
||||
|
||||
## Validation diagnostics
|
||||
|
||||
The validation diagnostics combine convergent and discriminant comparisons with expert surveys, posterior predictive coverage, construct checks, and out-of-sample validation when the corresponding outputs are available.
|
||||
|
||||
| diagnostic | dimension | n | pearson_r | spearman_r | mae | coverage |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| convergent validity | economic left-right | 4637 | 0.904 | 0.901 | 0.076 | NA |
|
||||
| convergent validity | cultural cosmopolitan--traditionalist | 1425 | 0.96 | 0.967 | 0.079 | NA |
|
||||
| posterior predictive coverage | economic left-right | 7455 | NA | NA | NA | 0.899 |
|
||||
| posterior predictive coverage | cultural cosmopolitan--traditionalist | 15539 | NA | NA | NA | 0.848 |
|
||||
|
||||
### Discriminant validity
|
||||
|
||||
| model_dim | expert_dim | r_pearson | r_spearman | n | type | diagnostic |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| economic left-right | economic | 0.904 | 0.901 | 4637 | convergent | discriminant validity |
|
||||
| cultural cosmopolitan--traditionalist | economic | 0.422 | 0.435 | 4637 | discriminant | discriminant validity |
|
||||
| cultural cosmopolitan--traditionalist | cultural cosmopolitan--traditionalist | 0.96 | 0.967 | 1425 | convergent | discriminant validity |
|
||||
| economic left-right | cultural cosmopolitan--traditionalist | 0.393 | 0.398 | 1425 | discriminant | discriminant validity |
|
||||
|
||||
### External validation correlations
|
||||
|
||||
| var | dimension | n | pearson_r | mean_absolute_error | coverage_95 |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| culsup_vparty | cultural cosmopolitan--traditionalist | 536 | 0.812 | 0.128 | 0.384 |
|
||||
| galtan_ches | cultural cosmopolitan--traditionalist | 222 | 0.959 | 0.079 | 0.523 |
|
||||
| gender_vparty | cultural cosmopolitan--traditionalist | 545 | 0.563 | 0.169 | 0.29 |
|
||||
| immig_vparty | cultural cosmopolitan--traditionalist | 537 | 0.743 | 0.103 | 0.49 |
|
||||
| lgbt_vparty | cultural cosmopolitan--traditionalist | 541 | 0.794 | 0.094 | 0.551 |
|
||||
| relig_vparty | cultural cosmopolitan--traditionalist | 548 | 0.676 | 0.303 | 0.047 |
|
||||
| lrecon_ches | economic left-right | 223 | 0.974 | 0.055 | 0.812 |
|
||||
| lrecon_poppa | economic left-right | 74 | 0.98 | 0.066 | 0.662 |
|
||||
| lrecon_vparty | economic left-right | 534 | 0.866 | 0.088 | 0.674 |
|
||||
| welf_vparty | economic left-right | 534 | 0.682 | 0.176 | 0.363 |
|
||||
|
||||
## Generated tables
|
||||
|
||||
- `alliance_union_harmonization.csv`
|
||||
- `construct_family_positions.csv`
|
||||
- `construct_temporal_stability_flags.csv`
|
||||
- `external_validation_correlations.csv`
|
||||
- `item_coding_orientation.csv`
|
||||
- `item_coverage.csv`
|
||||
- `model_convergence_by_dimension.csv`
|
||||
- `model_convergence_summary.csv`
|
||||
- `party_family_coverage.csv`
|
||||
- `party_year_source_coverage.csv`
|
||||
- `posterior_uncertainty_summary.csv`
|
||||
- `posterior_validation_convergent_summary.csv`
|
||||
- `posterior_validation_discriminant_summary.csv`
|
||||
- `posterior_validation_uncertainty_summary.csv`
|
||||
- `reversed_items.csv`
|
||||
- `source_coverage.csv`
|
||||
|
||||
Binary file not shown.
Reference in New Issue
Block a user