party2d/data-setup/README.md

Data setup

This directory documents and scripts the optional process for rebuilding the model-ready inputs from local raw/source datasets. The main estimation workflow does not run these scripts; once the five model-ready CSVs exist in data/, fitting and post-estimation are Julia/Stan-only.

The setup workflow is intentionally local-only. It never overwrites committed files in data/. Raw downloads, intermediate files, regenerated inputs, and comparison reports go under _local/, which is ignored by git.

Raw source files are not redistributed in this repository. Put them in _local/raw/ or set PARTY2D_RAW_DATA_DIR to another local directory. See source_manifest.csv and ../docs/RAW_DATA_SOURCES.md for source-specific access notes.

Recommended local layout:

_local/raw/
  manifesto/MPDataset_MPDS2025a.csv
  poldem/poldem-election_all.csv
  ches/...
  vparty/...
  poppa/...
  gps/...
  morgan/...
  partyfacts/partyfacts-external-parties.csv

Local output layout:

_local/build/             # intermediate processing files
_local/generated-inputs/  # regenerated final model-input CSVs
_local/reports/           # comparison reports

Check setup without downloading or rebuilding:

bash data-setup/run_data_setup.sh --dry-run

Attempt automatic downloads where permitted/available:

bash data-setup/run_data_setup.sh --download-only

The downloader handles public direct downloads, Harvard Dataverse files, and the V-Dem form workflow. For V-Dem, set an email address accepted by the provider's download form:

export PARTY2D_VDEM_EMAIL='you@example.org'
export PARTY2D_VDEM_GENDER=''   # blank means prefer not to say

The Manifesto Project main dataset is behind the Manifesto API/login terms. To download it through the script, set one of:

export MANIFESTO_API_KEY='...'
# or
export PARTY2D_MANIFESTO_API_KEY='...'

morgan/morgan_positions_raw.csv is a local OCR/transcription source derived from Morgan (1976), not a public provider download. Place it under _local/raw/morgan/ before a full rebuild test.

Rebuild model-ready inputs after placing all required local files. This writes to _local/generated-inputs/, not data/:

bash data-setup/run_data_setup.sh --build-test

Compare regenerated inputs with the committed inputs:

bash data-setup/run_data_setup.sh --compare

Run the full local test sequence — public downloads where available, raw-file preflight, local rebuild, and comparison — after manually placing restricted sources:

bash data-setup/run_data_setup.sh --full-test

The comparison writes _local/reports/input_comparison.md and exits nonzero if any generated input is missing or differs. Replacing committed inputs, if ever needed, is a separate manual decision and is not done by these scripts.