Raw data sources and local-only setup

This repository includes model-ready inputs under data/, but it does not redistribute licensed, restricted, or third-party raw source files. Raw files needed to regenerate processed inputs should be kept in _local/raw/ or another local-only directory selected with PARTY2D_RAW_DATA_DIR.

The source setup scripts are provided specifically to avoid redistributing those source files while still documenting how to obtain and test them locally. Users must use their own Manifesto Project API key and must obtain the Morgan OCR/ transcription file separately.

The data setup workflow writes intermediates to _local/build/, regenerated inputs to _local/generated-inputs/, and comparison reports to _local/reports/. It does not overwrite committed data/ files.

Recommended local layout:

_local/raw/
  poldem/poldem-election_all.csv
  manifesto/MPDataset_MPDS2025a.csv
  partyfacts/partyfacts-external-parties.csv
  ches/...
  vparty/...
  poppa/...
  gps/...
  morgan/...

The scripts read PARTY2D_RAW_DATA_DIR when it is set. For another location, use:

export PARTY2D_RAW_DATA_DIR=/path/to/local/raw

Required raw inputs

Source	Raw file	Local path below `$PARTY2D_RAW_DATA_DIR`	Used by	Redistribution
Manifesto Project Dataset	`MPDataset_MPDS2025a.csv`	`manifesto/MPDataset_MPDS2025a.csv`	`data-setup/R/process_manifesto.R`	Not redistributed in this repo
PolDem Election Campaigns, all issues	`poldem-election_all.csv`	`poldem/poldem-election_all.csv`	`data-setup/R/process_poldem.R`	Not redistributed in this repo
CHES family	CHES aggregate and expert-level files	`ches/`	`data-setup/R/process_expert.R`	Not redistributed in this repo
V-Party	`V-Dem-CPD-Party-V2.rds`	`vparty/V-Dem-CPD-Party-V2.rds`	`data-setup/R/process_expert.R`	Not redistributed in this repo
POPPA	`poppa_integrated_v2.rds`	`poppa/poppa_integrated_v2.rds`	`data-setup/R/process_expert.R`	Not redistributed in this repo
Global Party Survey 2019	`Global Party Survey by Party SPSS V2_1_Apr_2020-2.tab`	`gps/Global Party Survey by Party SPSS V2_1_Apr_2020-2.tab`	`data-setup/R/process_expert.R`	Not redistributed in this repo
Morgan historical expert data	`morgan_positions_raw.csv`	`morgan/morgan_positions_raw.csv`	`data-setup/R/process_morgan.R`	Not redistributed in this repo
PartyFacts crosswalk	`partyfacts-external-parties.csv`	`partyfacts/partyfacts-external-parties.csv`	source harmonization scripts	Not redistributed in this repo

PolDem download

Dataset: poldem-election_all from the PolDem Election Campaigns collection.

Overview: https://poldem.eui.eu/data-overview/
Download page: https://poldem.eui.eu/download/election-campaigns/
CSV URL used locally on 2026-06-11: https://poldem.eui.eu/downloads/cosa/poldem-election_all.csv

Observed checksum after download on 2026-06-11:

sha256  2cd8c9108b1b0b9c1b6594bb21acee709c70259cd02f450bc69fc09b505fc9fb

Preflight check

From the repository root, run:

bash data-setup/run_data_setup.sh --dry-run

This parses the setup scripts without downloading or rebuilding. To check local raw file placement and print byte sizes/checksums, run:

bash data-setup/check_raw_data.sh

After restricted/manual files are in place, run the full local rebuild and comparison test with:

bash data-setup/run_data_setup.sh --full-test

The generated files remain under _local/generated-inputs/. They are compared to the committed model-ready inputs, but never copied over them automatically.

Processed files kept in this repository

The committed files under data/ are limited to the model-ready inputs used by the Julia/Stan estimation path:

data/text_data.csv
data/expert.csv
data/lr_data.csv
data/union_mapping.csv
data/party_families.csv

These files document the analysis-ready inputs while avoiding redistribution of the underlying raw source data.

See data-setup/source_manifest.csv for a machine-readable source checklist.

Scripted download status

data-setup/download_sources.py downloads all sources that are script-accessible without project-specific credentials: PolDem, PartyFacts, CHES family files, POPPA from Harvard Dataverse, GPS from Harvard Dataverse, and V-Party through the provider form when PARTY2D_VDEM_EMAIL is set.

The Manifesto Project main dataset requires a Manifesto API key/login. Set MANIFESTO_API_KEY or PARTY2D_MANIFESTO_API_KEY for scripted download.

The Morgan historical file is a local OCR/transcription source from Morgan (1976), not a public provider download. It can be provided on request and must be placed locally at $PARTY2D_RAW_DATA_DIR/morgan/morgan_positions_raw.csv for full rebuild tests.

4.8 KiB Raw Blame History