armin/party2d
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Wire local-only data setup workflow

Browse Source
This commit is contained in:
aseimel
2026-06-15 14:38:47 +02:00
parent b5ca9370f1
commit d9c648bd54
21 changed files with 2442 additions and 110 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/RAW_DATA_SOURCES.md
+58 -13
View File
@@ -1,9 +1,13 @@
# Raw data sources and local-only setup
The public replication repository includes processed inputs under `data/`, but it
does not redistribute licensed or third-party raw source files. Raw files needed
to regenerate processed inputs should be kept in a local-only directory outside
the replication git repository.
This repository includes model-ready inputs under `data/`, but it does not
redistribute licensed, restricted, or third-party raw source files. Raw files
needed to regenerate processed inputs should be kept in `_local/raw/` or another
local-only directory selected with `PARTY2D_RAW_DATA_DIR`.
The data setup workflow writes intermediates to `_local/build/`, regenerated
inputs to `_local/generated-inputs/`, and comparison reports to `_local/reports/`.
It does not overwrite committed `data/` files.
Recommended local layout:
@@ -11,6 +15,12 @@ Recommended local layout:
_local/raw/
poldem/poldem-election_all.csv
manifesto/MPDataset_MPDS2025a.csv
partyfacts/partyfacts-external-parties.csv
ches/...
vparty/...
poppa/...
gps/...
morgan/...
```
The scripts read `PARTY2D_RAW_DATA_DIR` when it is set. For another location, use:
@@ -23,8 +33,14 @@ export PARTY2D_RAW_DATA_DIR=/path/to/local/raw
| Source | Raw file | Local path below `$PARTY2D_RAW_DATA_DIR` | Used by | Redistribution |
| --- | --- | --- | --- | --- |
| PolDem Election Campaigns, all issues | `poldem-election_all.csv` | `poldem/poldem-election_all.csv` | `src/r/00c_process_poldem.R` | Not redistributed in this repo |
| Manifesto Project Dataset | `MPDataset_MPDS2025a.csv` | `manifesto/MPDataset_MPDS2025a.csv` | `src/r/00a_process_manifesto.R` if `data/manifesto_data.csv` is regenerated | Not redistributed in this repo |
| Manifesto Project Dataset | `MPDataset_MPDS2025a.csv` | `manifesto/MPDataset_MPDS2025a.csv` | `data-setup/R/process_manifesto.R` | Not redistributed in this repo |
| PolDem Election Campaigns, all issues | `poldem-election_all.csv` | `poldem/poldem-election_all.csv` | `data-setup/R/process_poldem.R` | Not redistributed in this repo |
| CHES family | CHES aggregate and expert-level files | `ches/` | `data-setup/R/process_expert.R` | Not redistributed in this repo |
| V-Party | `V-Dem-CPD-Party-V2.rds` | `vparty/V-Dem-CPD-Party-V2.rds` | `data-setup/R/process_expert.R` | Not redistributed in this repo |
| POPPA | `poppa_integrated_v2.rds` | `poppa/poppa_integrated_v2.rds` | `data-setup/R/process_expert.R` | Not redistributed in this repo |
| Global Party Survey 2019 | `Global Party Survey by Party SPSS V2_1_Apr_2020-2.tab` | `gps/Global Party Survey by Party SPSS V2_1_Apr_2020-2.tab` | `data-setup/R/process_expert.R` | Not redistributed in this repo |
| Morgan historical expert data | `morgan_positions_raw.csv` | `morgan/morgan_positions_raw.csv` | `data-setup/R/process_morgan.R` | Not redistributed in this repo |
| PartyFacts crosswalk | `partyfacts-external-parties.csv` | `partyfacts/partyfacts-external-parties.csv` | source harmonization scripts | Not redistributed in this repo |
## PolDem download
@@ -46,22 +62,51 @@ sha256 2cd8c9108b1b0b9c1b6594bb21acee709c70259cd02f450bc69fc09b505fc9fb
From the repository root, run:
```bash
bash src/sh/check_raw_data.sh
bash data-setup/run_data_setup.sh --dry-run
```
This checks required local raw inputs and prints byte sizes and SHA-256 checksums.
This parses the setup scripts without downloading or rebuilding. To check local
raw file placement and print byte sizes/checksums, run:
```bash
bash data-setup/check_raw_data.sh
```
After restricted/manual files are in place, run the full local rebuild and comparison test with:
```bash
bash data-setup/run_data_setup.sh --full-test
```
The generated files remain under `_local/generated-inputs/`. They are compared to
the committed model-ready inputs, but never copied over them automatically.
## Processed files kept in this repository
The processed files under `data/` are intended to be part of this repository,
including:
The committed files under `data/` are limited to the model-ready inputs used by
the Julia/Stan estimation path:
- `data/poldem_data.csv`
- `data/manifesto_data.csv`
- `data/text_data.csv`
- `data/expert.csv`
- `data/lr_data.csv`
- `data/model_data.csv`
- `data/union_mapping.csv`
- `data/party_families.csv`
These files document the analysis-ready inputs while avoiding redistribution of
the underlying raw source data.
See `data-setup/source_manifest.csv` for a machine-readable source checklist.
## Scripted download status
`data-setup/download_sources.py` downloads all sources that are script-accessible
without project-specific credentials: PolDem, PartyFacts, CHES family files,
POPPA from Harvard Dataverse, GPS from Harvard Dataverse, and V-Party through the
provider form when `PARTY2D_VDEM_EMAIL` is set.
The Manifesto Project main dataset requires a Manifesto API key/login. Set
`MANIFESTO_API_KEY` or `PARTY2D_MANIFESTO_API_KEY` for scripted download.
The Morgan historical file is a local OCR/transcription source from Morgan
(1976), not a public provider download. It must be placed locally at
`$PARTY2D_RAW_DATA_DIR/morgan/morgan_positions_raw.csv` for full rebuild tests.
docs/union_mapping.md
+1 -1
View File
@@ -41,7 +41,7 @@ The V4 Stan model (`stan_model_2dim_v4.stan`) uses these mappings to produce **i
### Data flow
```
R pipeline (00_data-management.R):
Data setup pipeline (`data-setup/R/02_build_model_inputs.R`):
- text_data keeps union PF IDs (211) for manifesto rows
- expert/lr_data keeps individual PF IDs (1375, 1731)
- Expert filtering: party in text_data OR party in constituent_parties