aseimel
118 lines
4.7 KiB
Markdown
118 lines
4.7 KiB
Markdown
# Raw data sources and local-only setup
|
|
|
|
This repository includes model-ready inputs under `data/`, but it does not
|
|
redistribute licensed, restricted, or third-party raw source files. Raw files
|
|
needed to regenerate processed inputs should be kept in `_local/raw/` or another
|
|
directory selected with `PARTY2D_RAW_DATA_DIR`.
|
|
|
|
The source setup scripts document how to obtain and test those files locally.
|
|
Users must use their own Manifesto Project API key and obtain the Morgan OCR/
|
|
transcription file separately.
|
|
|
|
The data setup workflow writes intermediates to `_local/build/`, regenerated
|
|
inputs to `_local/generated-inputs/`, and comparison reports to `_local/reports/`.
|
|
It does not overwrite committed `data/` files.
|
|
|
|
Recommended local layout:
|
|
|
|
```text
|
|
_local/raw/
|
|
poldem/poldem-election_all.csv
|
|
manifesto/MPDataset_MPDS2025a.csv
|
|
partyfacts/partyfacts-external-parties.csv
|
|
ches/...
|
|
vparty/...
|
|
poppa/...
|
|
gps/...
|
|
morgan/...
|
|
```
|
|
|
|
The scripts read `PARTY2D_RAW_DATA_DIR` when it is set. For another location, use:
|
|
|
|
```bash
|
|
export PARTY2D_RAW_DATA_DIR=/path/to/local/raw
|
|
```
|
|
|
|
## Required raw inputs
|
|
|
|
| Source | Raw file | Local path below `$PARTY2D_RAW_DATA_DIR` | Used by | Redistribution |
|
|
| --- | --- | --- | --- | --- |
|
|
| Manifesto Project Dataset | `MPDataset_MPDS2025a.csv` | `manifesto/MPDataset_MPDS2025a.csv` | `data-setup/R/process_manifesto.R` | Not redistributed in this repo |
|
|
| PolDem Election Campaigns, all issues | `poldem-election_all.csv` | `poldem/poldem-election_all.csv` | `data-setup/R/process_poldem.R` | Not redistributed in this repo |
|
|
| CHES family | CHES aggregate and expert-level files | `ches/` | `data-setup/R/process_expert.R` | Not redistributed in this repo |
|
|
| V-Party | `V-Dem-CPD-Party-V2.rds` | `vparty/V-Dem-CPD-Party-V2.rds` | `data-setup/R/process_expert.R` | Not redistributed in this repo |
|
|
| POPPA | `poppa_integrated_v2.rds` | `poppa/poppa_integrated_v2.rds` | `data-setup/R/process_expert.R` | Not redistributed in this repo |
|
|
| Global Party Survey 2019 | `Global Party Survey by Party SPSS V2_1_Apr_2020-2.tab` | `gps/Global Party Survey by Party SPSS V2_1_Apr_2020-2.tab` | `data-setup/R/process_expert.R` | Not redistributed in this repo |
|
|
| Morgan historical expert data | `morgan_positions_raw.csv` | `morgan/morgan_positions_raw.csv` | `data-setup/R/process_morgan.R` | Not redistributed in this repo |
|
|
| PartyFacts crosswalk | `partyfacts-external-parties.csv` | `partyfacts/partyfacts-external-parties.csv` | source harmonization scripts | Not redistributed in this repo |
|
|
|
|
## PolDem download
|
|
|
|
Dataset: `poldem-election_all` from the PolDem Election Campaigns collection.
|
|
|
|
- Overview: <https://poldem.eui.eu/data-overview/>
|
|
- Download page: <https://poldem.eui.eu/download/election-campaigns/>
|
|
- CSV URL used locally on 2026-06-11:
|
|
<https://poldem.eui.eu/downloads/cosa/poldem-election_all.csv>
|
|
|
|
Observed checksum after download on 2026-06-11:
|
|
|
|
```text
|
|
sha256 2cd8c9108b1b0b9c1b6594bb21acee709c70259cd02f450bc69fc09b505fc9fb
|
|
```
|
|
|
|
## Preflight check
|
|
|
|
From the repository root, run:
|
|
|
|
```bash
|
|
bash data-setup/run_data_setup.sh --dry-run
|
|
```
|
|
|
|
This parses the setup scripts without downloading or rebuilding. To check local
|
|
raw file placement and print byte sizes/checksums, run:
|
|
|
|
```bash
|
|
bash data-setup/check_raw_data.sh
|
|
```
|
|
|
|
After restricted/manual files are in place, run the full local rebuild and comparison test with:
|
|
|
|
```bash
|
|
bash data-setup/run_data_setup.sh --full-test
|
|
```
|
|
|
|
The generated files remain under `_local/generated-inputs/`. They are compared to
|
|
the committed model-ready inputs, but never copied over them automatically.
|
|
|
|
## Processed files kept in this repository
|
|
|
|
The committed files under `data/` are limited to the model-ready inputs used by
|
|
the Julia/Stan estimation path:
|
|
|
|
- `data/text_data.csv`
|
|
- `data/expert.csv`
|
|
- `data/lr_data.csv`
|
|
- `data/union_mapping.csv`
|
|
- `data/party_families.csv`
|
|
|
|
These files document the analysis-ready inputs while avoiding redistribution of
|
|
the underlying raw source data.
|
|
|
|
See `data-setup/source_manifest.csv` for a machine-readable source checklist.
|
|
|
|
## Scripted download status
|
|
|
|
`data-setup/download_sources.py` downloads all sources that are script-accessible
|
|
without project-specific credentials: PolDem, PartyFacts, CHES family files,
|
|
POPPA from Harvard Dataverse, GPS from Harvard Dataverse, and V-Party through the
|
|
provider form when `PARTY2D_VDEM_EMAIL` is set.
|
|
|
|
The Manifesto Project main dataset requires a Manifesto API key/login. Set
|
|
`MANIFESTO_API_KEY` or `PARTY2D_MANIFESTO_API_KEY` for scripted download.
|
|
|
|
The Morgan historical file is a local OCR/transcription source from Morgan
|
|
(1976), not a public provider download. It can be provided on request and must be
|
|
placed locally at `$PARTY2D_RAW_DATA_DIR/morgan/morgan_positions_raw.csv` for
|
|
full rebuild tests.
|