Raw data sources and local-only setup

This repository includes model-ready inputs under data/, but it does not redistribute licensed, restricted, or third-party raw source files. Raw files needed to regenerate processed inputs should be kept in _local/raw/ or another local-only directory selected with PARTY2D_RAW_DATA_DIR.

The data setup workflow writes intermediates to _local/build/, regenerated inputs to _local/generated-inputs/, and comparison reports to _local/reports/. It does not overwrite committed data/ files.

Recommended local layout:

_local/raw/
  poldem/poldem-election_all.csv
  manifesto/MPDataset_MPDS2025a.csv
  partyfacts/partyfacts-external-parties.csv
  ches/...
  vparty/...
  poppa/...
  gps/...
  morgan/...

The scripts read PARTY2D_RAW_DATA_DIR when it is set. For another location, use:

export PARTY2D_RAW_DATA_DIR=/path/to/local/raw

Required raw inputs

Source	Raw file	Local path below `$PARTY2D_RAW_DATA_DIR`	Used by	Redistribution
Manifesto Project Dataset	`MPDataset_MPDS2025a.csv`	`manifesto/MPDataset_MPDS2025a.csv`	`data-setup/R/process_manifesto.R`	Not redistributed in this repo
PolDem Election Campaigns, all issues	`poldem-election_all.csv`	`poldem/poldem-election_all.csv`	`data-setup/R/process_poldem.R`	Not redistributed in this repo
CHES family	CHES aggregate and expert-level files	`ches/`	`data-setup/R/process_expert.R`	Not redistributed in this repo
V-Party	`V-Dem-CPD-Party-V2.rds`	`vparty/V-Dem-CPD-Party-V2.rds`	`data-setup/R/process_expert.R`	Not redistributed in this repo
POPPA	`poppa_integrated_v2.rds`	`poppa/poppa_integrated_v2.rds`	`data-setup/R/process_expert.R`	Not redistributed in this repo
Global Party Survey 2019	`Global Party Survey by Party SPSS V2_1_Apr_2020-2.tab`	`gps/Global Party Survey by Party SPSS V2_1_Apr_2020-2.tab`	`data-setup/R/process_expert.R`	Not redistributed in this repo
Morgan historical expert data	`morgan_positions_raw.csv`	`morgan/morgan_positions_raw.csv`	`data-setup/R/process_morgan.R`	Not redistributed in this repo
PartyFacts crosswalk	`partyfacts-external-parties.csv`	`partyfacts/partyfacts-external-parties.csv`	source harmonization scripts	Not redistributed in this repo

PolDem download

Dataset: poldem-election_all from the PolDem Election Campaigns collection.

Overview: https://poldem.eui.eu/data-overview/
Download page: https://poldem.eui.eu/download/election-campaigns/
CSV URL used locally on 2026-06-11: https://poldem.eui.eu/downloads/cosa/poldem-election_all.csv

Observed checksum after download on 2026-06-11:

sha256  2cd8c9108b1b0b9c1b6594bb21acee709c70259cd02f450bc69fc09b505fc9fb

Preflight check

From the repository root, run:

bash data-setup/run_data_setup.sh --dry-run

This parses the setup scripts without downloading or rebuilding. To check local raw file placement and print byte sizes/checksums, run:

bash data-setup/check_raw_data.sh

After restricted/manual files are in place, run the full local rebuild and comparison test with:

bash data-setup/run_data_setup.sh --full-test

The generated files remain under _local/generated-inputs/. They are compared to the committed model-ready inputs, but never copied over them automatically.

Processed files kept in this repository

The committed files under data/ are limited to the model-ready inputs used by the Julia/Stan estimation path:

data/text_data.csv
data/expert.csv
data/lr_data.csv
data/union_mapping.csv
data/party_families.csv

These files document the analysis-ready inputs while avoiding redistribution of the underlying raw source data.

See data-setup/source_manifest.csv for a machine-readable source checklist.

Scripted download status

data-setup/download_sources.py downloads all sources that are script-accessible without project-specific credentials: PolDem, PartyFacts, CHES family files, POPPA from Harvard Dataverse, GPS from Harvard Dataverse, and V-Party through the provider form when PARTY2D_VDEM_EMAIL is set.

The Manifesto Project main dataset requires a Manifesto API key/login. Set MANIFESTO_API_KEY or PARTY2D_MANIFESTO_API_KEY for scripted download.

The Morgan historical file is a local OCR/transcription source from Morgan (1976), not a public provider download. It must be placed locally at $PARTY2D_RAW_DATA_DIR/morgan/morgan_positions_raw.csv for full rebuild tests.

4.5 KiB Raw Blame History