party2d/data-setup/README.md

Data setup

This directory exists because the public repository cannot redistribute the original raw/source files. It downloads the sources that can be fetched automatically, checks user-provided restricted files, rebuilds model-ready inputs, and compares them with the committed files in ../data/.

The main estimation workflow does not run these scripts. Once the five model-ready CSVs exist in data/, fitting and post-estimation are Julia/Stan-only.

The setup workflow never overwrites committed files in data/.

Put raw source files in _local/raw/ or set PARTY2D_RAW_DATA_DIR to another local directory. See source_manifest.csv and ../docs/RAW_DATA_SOURCES.md for source-specific access notes.

What downloads automatically?

data-setup/download_sources.py automatically downloads all source files that are script-accessible under the providers' terms:

Source	Automatic?	Requirement
PolDem	Yes	none
PartyFacts crosswalk	Yes	none
CHES family files	Yes	none
POPPA	Yes	none; downloaded from Harvard Dataverse
Global Party Survey 2019	Yes	none; downloaded from Harvard Dataverse
V-Party	Yes	set PARTY2D_VDEM_EMAIL; optionally set PARTY2D_VDEM_GENDER
Manifesto Project	Yes, with credentials	set your own MANIFESTO_API_KEY or PARTY2D_MANIFESTO_API_KEY
Morgan historical expert data	No	place morgan_positions_raw.csv locally; available on request

Do not commit downloaded or user-provided source files.

Recommended local layout:

_local/raw/
  manifesto/MPDataset_MPDS2025a.csv
  poldem/poldem-election_all.csv
  ches/...
  vparty/...
  poppa/...
  gps/...
  morgan/...
  partyfacts/partyfacts-external-parties.csv

Local output layout:

_local/build/             # intermediate processing files
_local/generated-inputs/  # regenerated final model-input CSVs
_local/reports/           # comparison reports

Commands

Check setup without downloading or rebuilding:

bash data-setup/run_data_setup.sh --dry-run

Download all script-accessible sources. Manifesto requires an API key; V-Party requires an email for the provider form:

bash data-setup/run_data_setup.sh --download-only

V-Party:

export PARTY2D_VDEM_EMAIL='you@example.org'
export PARTY2D_VDEM_GENDER=''   # blank means prefer not to say

Manifesto Project:

export MANIFESTO_API_KEY='...'
# or
export PARTY2D_MANIFESTO_API_KEY='...'

Morgan is not a public provider download. The local OCR/transcription file can be provided on request and should be placed at:

_local/raw/morgan/morgan_positions_raw.csv

Rebuild model-ready inputs after placing all required local files:

bash data-setup/run_data_setup.sh --build-test

Compare regenerated inputs with the committed inputs:

bash data-setup/run_data_setup.sh --compare

Run the full local test sequence — source downloads, raw-file preflight, local rebuild, and comparison — after setting required environment variables and placing Morgan locally:

bash data-setup/run_data_setup.sh --full-test

The comparison writes _local/reports/input_comparison.md. Replacing committed inputs, if ever needed, is a separate manual decision and is not done by these scripts.

Known behavior: the current public/rebuilt sources run through the workflow successfully, but regenerated text_data.csv, expert.csv, and lr_data.csv are not byte-identical to the committed model-ready inputs because of source-version and linkage differences. The comparison report records those differences explicitly.