party2d/data-setup/README.md

Data setup

This directory exists because the public repository cannot redistribute the original raw/source files. It checks user-provided source files, rebuilds model-ready inputs, and compares them with the committed files in ../data/.

The main estimation workflow does not run these scripts. Once the five model-ready CSVs exist in data/, fitting and post-estimation are Julia/Stan-only.

The setup workflow never overwrites committed files in data/.

Put raw source files in _local/raw/ or set PARTY2D_RAW_DATA_DIR to another local directory. See source_manifest.csv and ../docs/RAW_DATA_SOURCES.md for source-specific access notes.

Source availability

The public repository does not include an automatic source downloader. Users should obtain source files directly from the providers under the providers' terms and place them under _local/raw/ or another directory selected with PARTY2D_RAW_DATA_DIR.

Source	Requirement
PolDem	obtain from provider
PartyFacts crosswalk	obtain from provider
CHES family files	obtain from provider
POPPA	obtain from Harvard Dataverse
Global Party Survey 2019	obtain from Harvard Dataverse
V-Party	obtain from provider
Manifesto Project	use your own Manifesto Project access/API key
Morgan historical expert data	place morgan_positions_raw.csv locally; available on request

Do not commit downloaded or user-provided source files.

Recommended local layout:

_local/raw/
  manifesto/MPDataset_MPDS2025a.csv
  poldem/poldem-election_all.csv
  ches/...
  vparty/...
  poppa/...
  gps/...
  morgan/...
  partyfacts/partyfacts-external-parties.csv

Local output layout:

_local/build/             # intermediate processing files
_local/generated-inputs/  # regenerated final model-input CSVs
_local/reports/           # comparison reports

Commands

Run the full source-data setup workflow with:

bash data-setup/run_data_setup.sh

Maintenance options:

• --dry-run — check setup scripts only.
• --build-test — rebuild model-ready inputs from existing local sources.
• --compare — compare regenerated inputs with committed data/ inputs.
• no option — rebuild and compare locally from existing source files.

Minimal check:

bash data-setup/run_data_setup.sh --dry-run

Morgan is not a public provider download. The local OCR/transcription file can be provided on request and should be placed at:

_local/raw/morgan/morgan_positions_raw.csv

Rebuild model-ready inputs:

bash data-setup/run_data_setup.sh --build-test

Compare regenerated inputs:

bash data-setup/run_data_setup.sh --compare

Full local source test:

bash data-setup/run_data_setup.sh

The comparison writes _local/reports/input_comparison.md. Replacing committed inputs, if ever needed, is a separate manual decision and is not done by these scripts.

Known behavior: the current public/rebuilt sources run through the workflow successfully, but regenerated text_data.csv, expert.csv, and lr_data.csv are not byte-identical to the committed model-ready inputs because of source-version and linkage differences. The comparison report records those differences explicitly.