party2d/data-setup/README.md

Data setup

This directory exists because the public repository cannot redistribute the original raw/source files. It downloads script-accessible sources, checks user-provided source files, rebuilds model-ready inputs, and compares them with the committed files in ../data/.

The main estimation workflow does not run these scripts. Once the five model-ready CSVs exist in data/, fitting and post-estimation are Julia/Stan-only.

The setup workflow never overwrites committed files in data/.

Put raw source files in _local/raw/ or set PARTY2D_RAW_DATA_DIR to another local directory. See source_manifest.csv and ../docs/RAW_DATA_SOURCES.md for source-specific access notes.

What downloads automatically?

data-setup/R/01_download_sources.R downloads source files that are script-accessible under the providers' terms and reports sources that require credentials or manual local files.

Source	Automatic?	Requirement
PolDem	Yes	none
PartyFacts crosswalk	Yes	none
CHES family files	Yes	none where provider links are live
POPPA	Yes	none; downloaded from Harvard Dataverse
Global Party Survey 2019	Yes	none; downloaded from Harvard Dataverse
V-Party	Yes, with provider form details	set PARTY2D_VDEM_EMAIL; optionally set PARTY2D_VDEM_GENDER
Manifesto Project	Yes, with credentials	set your own MANIFESTO_API_KEY or PARTY2D_MANIFESTO_API_KEY
Morgan historical expert data	No	place morgan_positions_raw.csv locally; available on request

Do not commit downloaded or user-provided source files.

Recommended local layout:

_local/raw/
  manifesto/MPDataset_MPDS2025a.csv
  poldem/poldem-election_all.csv
  ches/...
  vparty/...
  poppa/...
  gps/...
  morgan/...
  partyfacts/partyfacts-external-parties.csv

Local output layout:

_local/build/             # intermediate processing files
_local/generated-inputs/  # regenerated final model-input CSVs
_local/reports/           # comparison reports

Commands

Run the full source-data setup workflow with:

bash data-setup/run_data_setup.sh

The command downloads script-accessible sources, checks required local files, rebuilds model-ready inputs under _local/generated-inputs/, and writes a comparison report under _local/reports/. It never replaces committed files in data/.

V-Party:

export PARTY2D_VDEM_EMAIL='you@example.org'
export PARTY2D_VDEM_GENDER=''   # blank means prefer not to say

Manifesto Project:

export MANIFESTO_API_KEY='...'
# or
export PARTY2D_MANIFESTO_API_KEY='...'

Morgan is not a public provider download. The local OCR/transcription file can be provided on request and should be placed at:

_local/raw/morgan/morgan_positions_raw.csv

The comparison writes _local/reports/input_comparison.md. Replacing committed inputs, if ever needed, is a separate manual decision and is not done by these scripts.

Known behavior: the current public/rebuilt sources run through the workflow successfully, but regenerated text_data.csv, expert.csv, and lr_data.csv are not byte-identical to the committed model-ready inputs because of source-version and linkage differences. The comparison report records those differences explicitly.