aseimel
106 lines
4.2 KiB
Markdown
106 lines
4.2 KiB
Markdown
# Data setup
|
|
|
|
This directory exists because the public repository cannot redistribute the original raw/source files. It provides a reproducible, local-only workflow to download the source files that can be downloaded automatically, check user-provided restricted files, rebuild model-ready inputs, and compare the rebuilt inputs with the committed files in `../data/`.
|
|
|
|
The main estimation workflow does not run these scripts. Once the five model-ready CSVs exist in `data/`, fitting and post-estimation are Julia/Stan-only.
|
|
|
|
The setup workflow is intentionally local-only. It never overwrites committed files in `data/`. Raw downloads, intermediate files, regenerated inputs, and comparison reports go under `_local/`, which is ignored by git.
|
|
|
|
Raw source files are not redistributed in this repository. Put them in `_local/raw/` or set `PARTY2D_RAW_DATA_DIR` to another local directory. See `source_manifest.csv` and `../docs/RAW_DATA_SOURCES.md` for source-specific access notes.
|
|
|
|
## What downloads automatically?
|
|
|
|
`data-setup/download_sources.py` automatically downloads all source files that are script-accessible under the providers' terms:
|
|
|
|
| Source | Automatic? | Requirement |
|
|
| --- | --- | --- |
|
|
| PolDem | Yes | none |
|
|
| PartyFacts crosswalk | Yes | none |
|
|
| CHES family files | Yes | none |
|
|
| POPPA | Yes | none; downloaded from Harvard Dataverse |
|
|
| Global Party Survey 2019 | Yes | none; downloaded from Harvard Dataverse |
|
|
| V-Party | Yes | set `PARTY2D_VDEM_EMAIL`; optionally set `PARTY2D_VDEM_GENDER` |
|
|
| Manifesto Project | Yes, with credentials | set your own `MANIFESTO_API_KEY` or `PARTY2D_MANIFESTO_API_KEY` |
|
|
| Morgan historical expert data | No | place `morgan_positions_raw.csv` locally; available on request |
|
|
|
|
Do not commit any downloaded or user-provided source files. All default paths are under ignored `_local/` directories.
|
|
|
|
Recommended local layout:
|
|
|
|
```text
|
|
_local/raw/
|
|
manifesto/MPDataset_MPDS2025a.csv
|
|
poldem/poldem-election_all.csv
|
|
ches/...
|
|
vparty/...
|
|
poppa/...
|
|
gps/...
|
|
morgan/...
|
|
partyfacts/partyfacts-external-parties.csv
|
|
```
|
|
|
|
Local output layout:
|
|
|
|
```text
|
|
_local/build/ # intermediate processing files
|
|
_local/generated-inputs/ # regenerated final model-input CSVs
|
|
_local/reports/ # comparison reports
|
|
```
|
|
|
|
## Commands
|
|
|
|
Check setup without downloading or rebuilding:
|
|
|
|
```bash
|
|
bash data-setup/run_data_setup.sh --dry-run
|
|
```
|
|
|
|
Download all automatically downloadable sources. This also downloads Manifesto if you set a Manifesto API key and V-Party if you set a V-Dem form email:
|
|
|
|
```bash
|
|
bash data-setup/run_data_setup.sh --download-only
|
|
```
|
|
|
|
For V-Party, set an email address accepted by the provider's download form:
|
|
|
|
```bash
|
|
export PARTY2D_VDEM_EMAIL='you@example.org'
|
|
export PARTY2D_VDEM_GENDER='' # blank means prefer not to say
|
|
```
|
|
|
|
For Manifesto Project, users must use their own API key from their Manifesto Project account:
|
|
|
|
```bash
|
|
export MANIFESTO_API_KEY='...'
|
|
# or
|
|
export PARTY2D_MANIFESTO_API_KEY='...'
|
|
```
|
|
|
|
`morgan/morgan_positions_raw.csv` is a local OCR/transcription source derived from Morgan (1976), not a public provider download. It can be provided on request. Place it under `_local/raw/morgan/` before a full rebuild test:
|
|
|
|
```text
|
|
_local/raw/morgan/morgan_positions_raw.csv
|
|
```
|
|
|
|
Rebuild model-ready inputs after placing all required local files. This writes to `_local/generated-inputs/`, not `data/`:
|
|
|
|
```bash
|
|
bash data-setup/run_data_setup.sh --build-test
|
|
```
|
|
|
|
Compare regenerated inputs with the committed inputs:
|
|
|
|
```bash
|
|
bash data-setup/run_data_setup.sh --compare
|
|
```
|
|
|
|
Run the full local test sequence — source downloads, raw-file preflight, local rebuild, and comparison — after setting required environment variables and placing Morgan locally:
|
|
|
|
```bash
|
|
bash data-setup/run_data_setup.sh --full-test
|
|
```
|
|
|
|
The comparison writes `_local/reports/input_comparison.md`. Replacing committed inputs, if ever needed, is a separate manual decision and is not done by these scripts.
|
|
|
|
Known behavior: the current public/rebuilt sources run through the workflow successfully, but regenerated `text_data.csv`, `expert.csv`, and `lr_data.csv` are not byte-identical to the committed model-ready inputs because of source-version and linkage differences. The comparison report records those differences explicitly.
|