Wire local-only data setup workflow

2026-06-15 14:38:47 +02:00
parent b5ca9370f1
commit d9c648bd54
21 changed files with 2442 additions and 110 deletions
@@ -0,0 +1,83 @@
+# Data setup
+
+This directory documents and scripts the optional process for rebuilding the model-ready inputs from local raw/source datasets. The main estimation workflow does not run these scripts; once the five model-ready CSVs exist in `data/`, fitting and post-estimation are Julia/Stan-only.
+
+The setup workflow is intentionally local-only. It never overwrites committed files in `data/`. Raw downloads, intermediate files, regenerated inputs, and comparison reports go under `_local/`, which is ignored by git.
+
+Raw source files are not redistributed in this repository. Put them in `_local/raw/` or set `PARTY2D_RAW_DATA_DIR` to another local directory. See `source_manifest.csv` and `../docs/RAW_DATA_SOURCES.md` for source-specific access notes.
+
+Recommended local layout:
+
+```text
+_local/raw/
+  manifesto/MPDataset_MPDS2025a.csv
+  poldem/poldem-election_all.csv
+  ches/...
+  vparty/...
+  poppa/...
+  gps/...
+  morgan/...
+  partyfacts/partyfacts-external-parties.csv
+```
+
+Local output layout:
+
+```text
+_local/build/             # intermediate processing files
+_local/generated-inputs/  # regenerated final model-input CSVs
+_local/reports/           # comparison reports
+```
+
+Check setup without downloading or rebuilding:
+
+```bash
+bash data-setup/run_data_setup.sh --dry-run
+```
+
+Attempt automatic downloads where permitted/available:
+
+```bash
+bash data-setup/run_data_setup.sh --download-only
+```
+
+The downloader handles public direct downloads, Harvard Dataverse files, and the
+V-Dem form workflow. For V-Dem, set an email address accepted by the provider's
+download form:
+
+```bash
+export PARTY2D_VDEM_EMAIL='you@example.org'
+export PARTY2D_VDEM_GENDER=''   # blank means prefer not to say
+```
+
+The Manifesto Project main dataset is behind the Manifesto API/login terms. To
+download it through the script, set one of:
+
+```bash
+export MANIFESTO_API_KEY='...'
+# or
+export PARTY2D_MANIFESTO_API_KEY='...'
+```
+
+`morgan/morgan_positions_raw.csv` is a local OCR/transcription source derived
+from Morgan (1976), not a public provider download. Place it under
+`_local/raw/morgan/` before a full rebuild test.
+
+Rebuild model-ready inputs after placing all required local files. This writes to `_local/generated-inputs/`, not `data/`:
+
+```bash
+bash data-setup/run_data_setup.sh --build-test
+```
+
+Compare regenerated inputs with the committed inputs:
+
+```bash
+bash data-setup/run_data_setup.sh --compare
+```
+
+Run the full local test sequence — public downloads where available, raw-file preflight, local rebuild, and comparison — after manually placing restricted sources:
+
+```bash
+bash data-setup/run_data_setup.sh --full-test
+```
+
+The comparison writes `_local/reports/input_comparison.md` and exits nonzero if any generated input is missing or differs. Replacing committed inputs, if ever needed, is a separate manual decision and is not done by these scripts.