aseimel
84 lines
2.8 KiB
Markdown
84 lines
2.8 KiB
Markdown
# Data setup
|
|
|
|
This directory documents and scripts the optional process for rebuilding the model-ready inputs from local raw/source datasets. The main estimation workflow does not run these scripts; once the five model-ready CSVs exist in `data/`, fitting and post-estimation are Julia/Stan-only.
|
|
|
|
The setup workflow is intentionally local-only. It never overwrites committed files in `data/`. Raw downloads, intermediate files, regenerated inputs, and comparison reports go under `_local/`, which is ignored by git.
|
|
|
|
Raw source files are not redistributed in this repository. Put them in `_local/raw/` or set `PARTY2D_RAW_DATA_DIR` to another local directory. See `source_manifest.csv` and `../docs/RAW_DATA_SOURCES.md` for source-specific access notes.
|
|
|
|
Recommended local layout:
|
|
|
|
```text
|
|
_local/raw/
|
|
manifesto/MPDataset_MPDS2025a.csv
|
|
poldem/poldem-election_all.csv
|
|
ches/...
|
|
vparty/...
|
|
poppa/...
|
|
gps/...
|
|
morgan/...
|
|
partyfacts/partyfacts-external-parties.csv
|
|
```
|
|
|
|
Local output layout:
|
|
|
|
```text
|
|
_local/build/ # intermediate processing files
|
|
_local/generated-inputs/ # regenerated final model-input CSVs
|
|
_local/reports/ # comparison reports
|
|
```
|
|
|
|
Check setup without downloading or rebuilding:
|
|
|
|
```bash
|
|
bash data-setup/run_data_setup.sh --dry-run
|
|
```
|
|
|
|
Attempt automatic downloads where permitted/available:
|
|
|
|
```bash
|
|
bash data-setup/run_data_setup.sh --download-only
|
|
```
|
|
|
|
The downloader handles public direct downloads, Harvard Dataverse files, and the
|
|
V-Dem form workflow. For V-Dem, set an email address accepted by the provider's
|
|
download form:
|
|
|
|
```bash
|
|
export PARTY2D_VDEM_EMAIL='you@example.org'
|
|
export PARTY2D_VDEM_GENDER='' # blank means prefer not to say
|
|
```
|
|
|
|
The Manifesto Project main dataset is behind the Manifesto API/login terms. To
|
|
download it through the script, set one of:
|
|
|
|
```bash
|
|
export MANIFESTO_API_KEY='...'
|
|
# or
|
|
export PARTY2D_MANIFESTO_API_KEY='...'
|
|
```
|
|
|
|
`morgan/morgan_positions_raw.csv` is a local OCR/transcription source derived
|
|
from Morgan (1976), not a public provider download. Place it under
|
|
`_local/raw/morgan/` before a full rebuild test.
|
|
|
|
Rebuild model-ready inputs after placing all required local files. This writes to `_local/generated-inputs/`, not `data/`:
|
|
|
|
```bash
|
|
bash data-setup/run_data_setup.sh --build-test
|
|
```
|
|
|
|
Compare regenerated inputs with the committed inputs:
|
|
|
|
```bash
|
|
bash data-setup/run_data_setup.sh --compare
|
|
```
|
|
|
|
Run the full local test sequence — public downloads where available, raw-file preflight, local rebuild, and comparison — after manually placing restricted sources:
|
|
|
|
```bash
|
|
bash data-setup/run_data_setup.sh --full-test
|
|
```
|
|
|
|
The comparison writes `_local/reports/input_comparison.md` and exits nonzero if any generated input is missing or differs. Replacing committed inputs, if ever needed, is a separate manual decision and is not done by these scripts.
|