armin/party2d
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
3da2ab57ca7a49acba46250e367f753bcc6ca673
party2d/validation/logs/posterior_predictive_coverage.log
T

147 lines
6.3 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
============================================================
UNCERTAINTY VALIDATION: Posterior Predictive Coverage
============================================================
Following Claassen (2019) validation framework
Posterior predictive intervals account for both position
uncertainty AND observation-level measurement noise.
Using specified run directory: /projects/party4d/archive/party2d_replication/outputs/model_outputs/latest/run_2026-06-12_09-34-03
Loaded expert_dim.csv: 22994 observations
Unique rr values: 4261
Item indices (var_exp_dim): [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
Dimensions (dim_idx_exp): [1, 2]
Loading 4 chain files (selective columns)...
Need 8547 columns (4261 rr × 2 dims + 24 item params + 1 phi)
Found 8547/8547 columns in chains
Loading chain 1: chain_1.csv... 2000 samples, 39.3s
Loading chain 2: chain_2.csv... 2000 samples, 22.4s
Loading chain 3: chain_3.csv... 2000 samples, 24.5s
Loading chain 4: chain_4.csv... 2000 samples, 16.7s
Combined: 8000 total posterior draws
V5 detected: using Beta(phi*K*mu, phi*K*(1-mu)) with per-observation K
Computing posterior predictive coverage (95% level)
Expert observations: 22994
Posterior draws: 8000
Progress: 5.0% (1149 / 22994)
Progress: 10.0% (2298 / 22994)
Progress: 15.0% (3447 / 22994)
Progress: 20.0% (4596 / 22994)
Progress: 25.0% (5745 / 22994)
Progress: 30.0% (6894 / 22994)
Progress: 35.0% (8043 / 22994)
Progress: 40.0% (9192 / 22994)
Progress: 45.0% (10341 / 22994)
Progress: 50.0% (11490 / 22994)
Progress: 55.0% (12639 / 22994)
Progress: 60.0% (13788 / 22994)
Progress: 65.0% (14937 / 22994)
Progress: 70.0% (16086 / 22994)
Progress: 75.0% (17235 / 22994)
Progress: 80.0% (18384 / 22994)
Progress: 84.9% (19533 / 22994)
Progress: 89.9% (20682 / 22994)
Progress: 94.9% (21831 / 22994)
Progress: 99.9% (22980 / 22994)
Progress: 100.0% (22994 / 22994)
============================================================
POSTERIOR PREDICTIVE COVERAGE (95%)
============================================================
economic left-right : 89.8% [89.1%, 90.5%] (6698/7455)
By survey source:
Project N PPC
V-Party 5645 87.5%
CHES 1177 97.3%
POPPA 384 98.7%
GPS 249 94.0%
By decade:
Decade N PPC
1970 658 88.1%
1980 712 88.9%
1990 1454 88.0%
2000 1758 89.4%
2010 2407 90.2%
2020 466 99.1%
cultural cosmopolitan--traditionalist : 84.8% [84.2%, 85.3%] (13170/15539)
By survey source:
Project N PPC
V-Party 14114 83.8%
CHES 1176 94.3%
GPS 249 91.2%
By decade:
Decade N PPC
1970 1633 86.2%
1980 1772 87.9%
1990 3485 85.5%
2000 3965 84.3%
2010 4426 82.1%
2020 258 97.7%
Saved detailed posterior-predictive results to: revision/outputs/ppc
============================================================
RECOMPUTING WITH 80% LEVEL (Claassen comparison)
============================================================
V5 detected: using Beta(phi*K*mu, phi*K*(1-mu)) with per-observation K
Computing posterior predictive coverage (80% level)
Expert observations: 22994
Posterior draws: 8000
Progress: 5.0% (1149 / 22994)
Progress: 10.0% (2298 / 22994)
Progress: 15.0% (3447 / 22994)
Progress: 20.0% (4596 / 22994)
Progress: 25.0% (5745 / 22994)
Progress: 30.0% (6894 / 22994)
Progress: 35.0% (8043 / 22994)
Progress: 40.0% (9192 / 22994)
Progress: 45.0% (10341 / 22994)
Progress: 50.0% (11490 / 22994)
Progress: 55.0% (12639 / 22994)
Progress: 60.0% (13788 / 22994)
Progress: 65.0% (14937 / 22994)
Progress: 70.0% (16086 / 22994)
Progress: 75.0% (17235 / 22994)
Progress: 80.0% (18384 / 22994)
Progress: 84.9% (19533 / 22994)
Progress: 89.9% (20682 / 22994)
Progress: 94.9% (21831 / 22994)
Progress: 99.9% (22980 / 22994)
Progress: 100.0% (22994 / 22994)
============================================================
POSTERIOR PREDICTIVE COVERAGE (80%)
============================================================
economic left-right : 75.9% [74.9%, 76.8%] (5655/7455)
By survey source:
Project N PPC
V-Party 5645 71.5%
CHES 1177 90.1%
POPPA 384 92.2%
GPS 249 82.7%
By decade:
Decade N PPC
1970 658 69.3%
1980 712 73.5%
1990 1454 73.8%
2000 1758 74.1%
2010 2407 77.0%
2020 466 95.7%
cultural cosmopolitan--traditionalist : 67.9% [67.1%, 68.6%] (10544/15539)
By survey source:
Project N PPC
V-Party 14114 66.3%
CHES 1176 83.9%
GPS 249 77.5%
By decade:
Decade N PPC
1970 1633 68.4%
1980 1772 69.8%
1990 3485 67.5%
2000 3965 68.6%
2010 4426 65.0%
2020 258 94.2%
Saved: revision/outputs/ppc/uncertainty_cic_summary_2026-08-12_16-50-24.csv
Saved: revision/outputs/ppc/uncertainty_cic_80pct_summary_2026-08-12_16-50-24.csv
Saved: revision/outputs/ppc/uncertainty_galtan_by_project_2026-08-12_16-50-24.csv
Saved: revision/outputs/ppc/uncertainty_economic_lr_by_project_2026-08-12_16-50-24.csv
============================================================
COMPARISON WITH CLAASSEN (2019) BENCHMARKS
============================================================
Claassen's result:
CIC (80% CI): 60.3%
(Using credible intervals for θ, not posterior predictive)
Our results (posterior predictive):
------------------------------------------------------------
Dimension PPC 95% PPC 80% Status
------------------------------------------------------------
economic_lr 89.8% 75.9% OK
galtan 84.8% 67.9% OK
------------------------------------------------------------
Interpretation:
95% PPC ~95% = well-calibrated uncertainty
80% PPC > 60% = exceeds Claassen (2019) benchmark
============================================================
VALIDATION COMPLETE
============================================================
Reference in New Issue View Git Blame Copy Permalink