Publish party-position estimates and validation materials

README.md

@@ -0,0 +1,156 @@
+# party2d
+
+Code and processed model inputs for generating two-dimensional party-position estimates from text and expert data. The model combines manifesto and media text indicators with expert survey placements in a Bayesian dynamic item-response framework.
+
+## Repository contents
+
+- `data-setup/` — source download, source-file checks, and rebuild workflow for raw files that cannot be redistributed here.
+- `src/julia/` — Stan data preparation, model fitting, post-estimation, enrichment, and validation.
+- `models/` — Stan model specification.
+- `data/` — processed party-level inputs used by the Julia/Stan model.
+- `metadata/` — data dictionary and source-support documentation.
+- `diagnostics/` — repository diagnostics report regenerated after model estimation.
+- `data/releases/` — release-ready data files, checksums, and diagnostics report.
+
+Processed inputs needed by the model are included in `data/` so the estimation step can be reproduced from the model-ready data.
+
+## Release assets
+
+The public `v0` release is available from <https://git.seimel.app/armin/party2d/releases> and contains:
+
+- `party_2d_election_year_panel_v0.zip` — primary election-year party-position panel.
+- `party_2d_annual_model_output_v0.csv.xz` — secondary annual model output (XZ-compressed CSV).
+- `party_2d_diagnostics_report_v0.pdf` — script-generated diagnostics report.
+- `SHA256SUMS` — checksums for the release assets.
+
+## Source data and redistribution
+
+The public repository does **not** contain the original raw/source files. Several inputs are third-party datasets with their own terms of use, and the Morgan historical file is a local OCR/transcription source. Instead, this repository provides:
+
+1. committed model-ready inputs in `data/`, sufficient for fitting the Julia/Stan model; and
+2. `data-setup/`, an R/Shell workflow that downloads script-accessible sources, checks locally supplied raw sources, and rebuilds comparable model-ready inputs locally.
+
+`data-setup/` downloads script-accessible sources and checks locally supplied sources for:
+
+- PolDem
+- PartyFacts crosswalk
+- CHES family files
+- POPPA from Harvard Dataverse
+- Global Party Survey 2019 from Harvard Dataverse
+- V-Party through the provider's download form
+
+Two inputs require user-provided access/material:
+
+- **Manifesto Project**: users must obtain the source data through their own Manifesto Project access.
+- **Morgan historical expert data**: `morgan_positions_raw.csv` is not publicly downloadable; it can be provided on request and should be placed locally under `_local/raw/morgan/`.
+
+The setup workflow never overwrites committed files in `data/`. See `data-setup/README.md` for exact commands and source details.
+
+Run the full source-data setup workflow with:
+
+```bash
+bash data-setup/run_data_setup.sh
+```
+
+This downloads script-accessible source files, checks required local files, rebuilds model-ready inputs locally, and writes a comparison report. Manifesto Project requires your own provider access, and the Morgan OCR/transcription file can be provided on request.
+
+## Running the pipeline
+
+Run the full workflow with:
+
+```bash
+bash run_estimation.sh full
+```
+
+This checks that model-ready inputs are present, then executes the Julia/Stan workflow scripts:
+
+```bash
+bash scripts/01_prepare_data.sh   # checks model-ready inputs; does not rebuild raw data
+bash scripts/02_fit_model.sh
+bash scripts/03_extract_estimates.sh
+bash scripts/04_enrich_estimates.sh
+bash scripts/05_validate_estimates.sh
+```
+
+The numbered scripts can also be run manually in that order.
+
+The Bayesian model is computationally expensive. The production run used 4 cores on an AMD Ryzen 9 7945HX and took 60,372 seconds, approximately 16 hours 46 minutes.
+
+If model output is already available, rebuild estimates without refitting Stan:
+
+```bash
+bash run_estimation.sh reuse
+```
+
+`reuse` verifies the model-ready inputs, then reruns post-estimation, enrichment, and validation while skipping the Stan fitting step.
+
+To check the local setup without fitting the model, run:
+
+```bash
+bash run_estimation.sh dry-run
+```
+
+After estimation and validation have been run, regenerate the diagnostics report with:
+
+```bash
+Rscript diagnostics/generate_diagnostics.R
+```
+
+The report is written to `diagnostics/generated/diagnostics_report.pdf` and copied to `data/releases/party_2d_diagnostics_report_v0.pdf`.
+
+## Data inputs
+
+The model-ready inputs are included under `data/`:
+
+- `text_data.csv`
+- `expert.csv`
+- `lr_data.csv`
+- `union_mapping.csv`
+- `party_families.csv`
+
+Original raw source files are not redistributed. Rebuilding inputs from raw files is separate from the normal estimation workflow and never replaces committed `data/` inputs automatically.
+
+## Output dimensions
+
+The two position dimensions are scaled from 0 to 1:
+
+- Economic left-right: economic left to economic right.
+- Cultural cosmopolitan--traditionalist: cosmopolitan to traditionalist.
+
+Column definitions are in `metadata/data_dictionary.csv`.
+
+## Release files
+
+The public release assets are:
+
+- `party_2d_election_year_panel_v0.zip`
+- `party_2d_annual_model_output_v0.csv.xz`
+- `party_2d_diagnostics_report_v0.pdf`
+- `SHA256SUMS`
+
+`SHA256SUMS` hashes the release assets listed above.
+
+## Public-content safeguard
+
+This repository is intended only for publication-ready material. Before pushing,
+run:
+
+```bash
+bash scripts/check_public_content.sh
+```
+
+To install the same audit as a local pre-push hook, run:
+
+```bash
+bash scripts/install_public_guard.sh
+```
+
+The audit checks the working public history and tracked files for non-public
+material. If it fails, do not push until the issue is resolved.
+
+## License
+
+This release uses the Creative Commons Attribution 4.0 International License
+(CC BY 4.0). The license applies to release materials created for this
+repository; third-party source data remain governed by their own terms. See
+`LICENSE`.