# Data setup

This directory documents and scripts the optional process for rebuilding the model-ready inputs from local raw/source datasets. The main estimation workflow does not run these scripts; once the five model-ready CSVs exist in `data/`, fitting and post-estimation are Julia/Stan-only.

The setup workflow is intentionally local-only. It never overwrites committed files in `data/`. Raw downloads, intermediate files, regenerated inputs, and comparison reports go under `_local/`, which is ignored by git.

Raw source files are not redistributed in this repository. Put them in `_local/raw/` or set `PARTY2D_RAW_DATA_DIR` to another local directory. See `source_manifest.csv` and `../docs/RAW_DATA_SOURCES.md` for source-specific access notes.

Recommended local layout:

```text
_local/raw/
  manifesto/MPDataset_MPDS2025a.csv
  poldem/poldem-election_all.csv
  ches/...
  vparty/...
  poppa/...
  gps/...
  morgan/...
  partyfacts/partyfacts-external-parties.csv
```

Local output layout:

```text
_local/build/             # intermediate processing files
_local/generated-inputs/  # regenerated final model-input CSVs
_local/reports/           # comparison reports
```

Check setup without downloading or rebuilding:

```bash
bash data-setup/run_data_setup.sh --dry-run
```

Attempt automatic downloads where permitted/available:

```bash
bash data-setup/run_data_setup.sh --download-only
```

The downloader handles public direct downloads, Harvard Dataverse files, and the
V-Dem form workflow. For V-Dem, set an email address accepted by the provider's
download form:

```bash
export PARTY2D_VDEM_EMAIL='you@example.org'
export PARTY2D_VDEM_GENDER=''   # blank means prefer not to say
```

The Manifesto Project main dataset is behind the Manifesto API/login terms. To
download it through the script, set one of:

```bash
export MANIFESTO_API_KEY='...'
# or
export PARTY2D_MANIFESTO_API_KEY='...'
```

`morgan/morgan_positions_raw.csv` is a local OCR/transcription source derived
from Morgan (1976), not a public provider download. Place it under
`_local/raw/morgan/` before a full rebuild test.

Rebuild model-ready inputs after placing all required local files. This writes to `_local/generated-inputs/`, not `data/`:

```bash
bash data-setup/run_data_setup.sh --build-test
```

Compare regenerated inputs with the committed inputs:

```bash
bash data-setup/run_data_setup.sh --compare
```

Run the full local test sequence — public downloads where available, raw-file preflight, local rebuild, and comparison — after manually placing restricted sources:

```bash
bash data-setup/run_data_setup.sh --full-test
```

The comparison writes `_local/reports/input_comparison.md` and exits nonzero if any generated input is missing or differs. Replacing committed inputs, if ever needed, is a separate manual decision and is not done by these scripts.