Armin
98 lines
3.2 KiB
Markdown
98 lines
3.2 KiB
Markdown
# Data setup
|
|
|
|
This directory exists because the public repository cannot redistribute the original raw/source files. It checks user-provided source files, rebuilds model-ready inputs, and compares them with the committed files in `../data/`.
|
|
|
|
The main estimation workflow does not run these scripts. Once the five model-ready CSVs exist in `data/`, fitting and post-estimation are Julia/Stan-only.
|
|
|
|
The setup workflow never overwrites committed files in `data/`.
|
|
|
|
Put raw source files in `_local/raw/` or set `PARTY2D_RAW_DATA_DIR` to another local directory. See `source_manifest.csv` and `../docs/RAW_DATA_SOURCES.md` for source-specific access notes.
|
|
|
|
## Source availability
|
|
|
|
The public repository does not include an automatic source downloader. Users should obtain source files directly from the providers under the providers' terms and place them under `_local/raw/` or another directory selected with `PARTY2D_RAW_DATA_DIR`.
|
|
|
|
| Source | Requirement |
|
|
| --- | --- |
|
|
| PolDem | obtain from provider |
|
|
| PartyFacts crosswalk | obtain from provider |
|
|
| CHES family files | obtain from provider |
|
|
| POPPA | obtain from Harvard Dataverse |
|
|
| Global Party Survey 2019 | obtain from Harvard Dataverse |
|
|
| V-Party | obtain from provider |
|
|
| Manifesto Project | use your own Manifesto Project access/API key |
|
|
| Morgan historical expert data | place `morgan_positions_raw.csv` locally; available on request |
|
|
|
|
Do not commit downloaded or user-provided source files.
|
|
|
|
Recommended local layout:
|
|
|
|
```text
|
|
_local/raw/
|
|
manifesto/MPDataset_MPDS2025a.csv
|
|
poldem/poldem-election_all.csv
|
|
ches/...
|
|
vparty/...
|
|
poppa/...
|
|
gps/...
|
|
morgan/...
|
|
partyfacts/partyfacts-external-parties.csv
|
|
```
|
|
|
|
Local output layout:
|
|
|
|
```text
|
|
_local/build/ # intermediate processing files
|
|
_local/generated-inputs/ # regenerated final model-input CSVs
|
|
_local/reports/ # comparison reports
|
|
```
|
|
|
|
## Commands
|
|
|
|
Run the full source-data setup workflow with:
|
|
|
|
```bash
|
|
bash data-setup/run_data_setup.sh
|
|
```
|
|
|
|
Maintenance options:
|
|
|
|
- `--dry-run` — check setup scripts only.
|
|
- `--build-test` — rebuild model-ready inputs from existing local sources.
|
|
- `--compare` — compare regenerated inputs with committed `data/` inputs.
|
|
- no option — rebuild and compare locally from existing source files.
|
|
|
|
Minimal check:
|
|
|
|
```bash
|
|
bash data-setup/run_data_setup.sh --dry-run
|
|
```
|
|
|
|
Morgan is not a public provider download. The local OCR/transcription file can be provided on request and should be placed at:
|
|
|
|
```text
|
|
_local/raw/morgan/morgan_positions_raw.csv
|
|
```
|
|
|
|
Rebuild model-ready inputs:
|
|
|
|
```bash
|
|
bash data-setup/run_data_setup.sh --build-test
|
|
```
|
|
|
|
Compare regenerated inputs:
|
|
|
|
```bash
|
|
bash data-setup/run_data_setup.sh --compare
|
|
```
|
|
|
|
Full local source test:
|
|
|
|
```bash
|
|
bash data-setup/run_data_setup.sh
|
|
```
|
|
|
|
The comparison writes `_local/reports/input_comparison.md`. Replacing committed inputs, if ever needed, is a separate manual decision and is not done by these scripts.
|
|
|
|
Known behavior: the current public/rebuilt sources run through the workflow successfully, but regenerated `text_data.csv`, `expert.csv`, and `lr_data.csv` are not byte-identical to the committed model-ready inputs because of source-version and linkage differences. The comparison report records those differences explicitly.
|