Raw data sources and local-only setup

This repository includes model-ready inputs under data/, but it does not redistribute licensed, restricted, or third-party raw source files. Raw files needed to regenerate processed inputs should be kept in _local/raw/ or another directory selected with PARTY2D_RAW_DATA_DIR.

The source setup scripts document how to obtain and test those files locally. Users must use their own Manifesto Project API key and obtain the Morgan OCR/ transcription file separately.

The data setup workflow writes intermediates to _local/build/, regenerated inputs to _local/generated-inputs/, and comparison reports to _local/reports/. It does not overwrite committed data/ files.

Recommended local layout:

_local/raw/
  poldem/poldem-election_all.csv
  manifesto/MPDataset_MPDS2025a.csv
  partyfacts/partyfacts-external-parties.csv
  ches/...
  vparty/...
  poppa/...
  gps/...
  morgan/...

The scripts read PARTY2D_RAW_DATA_DIR when it is set. For another location, use:

export PARTY2D_RAW_DATA_DIR=/path/to/local/raw

Required raw inputs

Source	Raw file	Local path below `$PARTY2D_RAW_DATA_DIR`	Used by	Redistribution
Manifesto Project Dataset	`MPDataset_MPDS2025a.csv`	`manifesto/MPDataset_MPDS2025a.csv`	`data-setup/R/process_manifesto.R`	Not redistributed in this repo
PolDem Election Campaigns, all issues	`poldem-election_all.csv`	`poldem/poldem-election_all.csv`	`data-setup/R/process_poldem.R`	Not redistributed in this repo
CHES family	CHES aggregate and expert-level files	`ches/`	`data-setup/R/process_expert.R`	Not redistributed in this repo
V-Party	`V-Dem-CPD-Party-V2.rds`	`vparty/V-Dem-CPD-Party-V2.rds`	`data-setup/R/process_expert.R`	Not redistributed in this repo
POPPA	`poppa_integrated_v2.rds`	`poppa/poppa_integrated_v2.rds`	`data-setup/R/process_expert.R`	Not redistributed in this repo
Global Party Survey 2019	`Global Party Survey by Party SPSS V2_1_Apr_2020-2.tab`	`gps/Global Party Survey by Party SPSS V2_1_Apr_2020-2.tab`	`data-setup/R/process_expert.R`	Not redistributed in this repo
Morgan historical expert data	`morgan_positions_raw.csv`	`morgan/morgan_positions_raw.csv`	`data-setup/R/process_morgan.R`	Not redistributed in this repo
PartyFacts crosswalk	`partyfacts-external-parties.csv`	`partyfacts/partyfacts-external-parties.csv`	source harmonization scripts	Not redistributed in this repo

PolDem download

Dataset: poldem-election_all from the PolDem Election Campaigns collection.

Overview: https://poldem.eui.eu/data-overview/
Download page: https://poldem.eui.eu/download/election-campaigns/
CSV URL used locally on 2026-06-11: https://poldem.eui.eu/downloads/cosa/poldem-election_all.csv

Observed checksum after download on 2026-06-11:

sha256  2cd8c9108b1b0b9c1b6594bb21acee709c70259cd02f450bc69fc09b505fc9fb

Preflight check

From the repository root, run:

bash data-setup/run_data_setup.sh --dry-run

To check local raw file placement and print byte sizes/checksums, run:

bash data-setup/check_raw_data.sh

After restricted/manual files are in place, run the full local rebuild and comparison test with:

bash data-setup/run_data_setup.sh

The generated files remain under _local/generated-inputs/. They are compared to the committed model-ready inputs, but never copied over them automatically.

Processed files kept in this repository

The committed files under data/ are limited to the model-ready inputs used by the Julia/Stan estimation path:

data/text_data.csv
data/expert.csv
data/lr_data.csv
data/union_mapping.csv
data/party_families.csv

These files document the analysis-ready inputs while avoiding redistribution of the underlying raw source data.

See data-setup/source_manifest.csv for a machine-readable source checklist.

Source access status

The public repository does not include an automatic source downloader. Obtain source files directly from the providers under their terms and place them in the documented local raw-data layout before running the rebuild checks.

The Manifesto Project main dataset requires Manifesto Project access/API credentials from the provider.

The Morgan historical file is a local OCR/transcription source from Morgan (1976), not a public provider download. It can be provided on request and must be placed locally at $PARTY2D_RAW_DATA_DIR/morgan/morgan_positions_raw.csv for full rebuild tests.

4.5 KiB Raw Blame History