# Data setup

This directory exists because the public repository cannot redistribute the original raw/source files. It downloads the sources that can be fetched automatically, checks user-provided restricted files, rebuilds model-ready inputs, and compares them with the committed files in `../data/`.

The main estimation workflow does not run these scripts. Once the five model-ready CSVs exist in `data/`, fitting and post-estimation are Julia/Stan-only.

The setup workflow never overwrites committed files in `data/`.

Put raw source files in `_local/raw/` or set `PARTY2D_RAW_DATA_DIR` to another local directory. See `source_manifest.csv` and `../docs/RAW_DATA_SOURCES.md` for source-specific access notes.

## What downloads automatically?

`data-setup/download_sources.py` automatically downloads all source files that are script-accessible under the providers' terms:

| Source | Automatic? | Requirement |
| --- | --- | --- |
| PolDem | Yes | none |
| PartyFacts crosswalk | Yes | none |
| CHES family files | Yes | none |
| POPPA | Yes | none; downloaded from Harvard Dataverse |
| Global Party Survey 2019 | Yes | none; downloaded from Harvard Dataverse |
| V-Party | Yes | set `PARTY2D_VDEM_EMAIL`; optionally set `PARTY2D_VDEM_GENDER` |
| Manifesto Project | Yes, with credentials | set your own `MANIFESTO_API_KEY` or `PARTY2D_MANIFESTO_API_KEY` |
| Morgan historical expert data | No | place `morgan_positions_raw.csv` locally; available on request |

Do not commit downloaded or user-provided source files.

Recommended local layout:

```text
_local/raw/
  manifesto/MPDataset_MPDS2025a.csv
  poldem/poldem-election_all.csv
  ches/...
  vparty/...
  poppa/...
  gps/...
  morgan/...
  partyfacts/partyfacts-external-parties.csv
```

Local output layout:

```text
_local/build/             # intermediate processing files
_local/generated-inputs/  # regenerated final model-input CSVs
_local/reports/           # comparison reports
```

## Commands

Run the setup script with one explicit option:

```bash
bash data-setup/run_data_setup.sh <option>
```

Available options:

- `--dry-run` — check setup scripts only.
- `--download-only` — download script-accessible sources.
- `--build-test` — rebuild model-ready inputs from existing local sources.
- `--compare` — compare regenerated inputs with committed `data/` inputs.
- `--full-test` — download, rebuild, and compare locally.

Minimal check:

```bash
bash data-setup/run_data_setup.sh --dry-run
```

Download sources:

```bash
bash data-setup/run_data_setup.sh --download-only
```

V-Party:

```bash
export PARTY2D_VDEM_EMAIL='you@example.org'
export PARTY2D_VDEM_GENDER=''   # blank means prefer not to say
```

Manifesto Project:

```bash
export MANIFESTO_API_KEY='...'
# or
export PARTY2D_MANIFESTO_API_KEY='...'
```

Morgan is not a public provider download. The local OCR/transcription file can be provided on request and should be placed at:

```text
_local/raw/morgan/morgan_positions_raw.csv
```

Rebuild model-ready inputs:

```bash
bash data-setup/run_data_setup.sh --build-test
```

Compare regenerated inputs:

```bash
bash data-setup/run_data_setup.sh --compare
```

Full local source test:

```bash
bash data-setup/run_data_setup.sh --full-test
```

The comparison writes `_local/reports/input_comparison.md`. Replacing committed inputs, if ever needed, is a separate manual decision and is not done by these scripts.

Known behavior: the current public/rebuilt sources run through the workflow successfully, but regenerated `text_data.csv`, `expert.csv`, and `lr_data.csv` are not byte-identical to the committed model-ready inputs because of source-version and linkage differences. The comparison report records those differences explicitly.