# Raw data sources and local-only setup

This repository includes model-ready inputs under `data/`, but it does not
redistribute licensed, restricted, or third-party raw source files. Raw files
needed to regenerate processed inputs should be kept in `_local/raw/` or another
directory selected with `PARTY2D_RAW_DATA_DIR`.

The source setup scripts document how to obtain and test those files locally.
Users must use their own Manifesto Project API key and obtain the Morgan OCR/
transcription file separately.

The data setup workflow writes intermediates to `_local/build/`, regenerated
inputs to `_local/generated-inputs/`, and comparison reports to `_local/reports/`.
It does not overwrite committed `data/` files.

Recommended local layout:

```text
_local/raw/
  poldem/poldem-election_all.csv
  manifesto/MPDataset_MPDS2025a.csv
  partyfacts/partyfacts-external-parties.csv
  ches/...
  vparty/...
  poppa/...
  gps/...
  morgan/...
```

The scripts read `PARTY2D_RAW_DATA_DIR` when it is set. For another location, use:

```bash
export PARTY2D_RAW_DATA_DIR=/path/to/local/raw
```

## Required raw inputs

| Source | Raw file | Local path below `$PARTY2D_RAW_DATA_DIR` | Used by | Redistribution |
| --- | --- | --- | --- | --- |
| Manifesto Project Dataset | `MPDataset_MPDS2025a.csv` | `manifesto/MPDataset_MPDS2025a.csv` | `data-setup/R/process_manifesto.R` | Not redistributed in this repo |
| PolDem Election Campaigns, all issues | `poldem-election_all.csv` | `poldem/poldem-election_all.csv` | `data-setup/R/process_poldem.R` | Not redistributed in this repo |
| CHES family | CHES aggregate and expert-level files | `ches/` | `data-setup/R/process_expert.R` | Not redistributed in this repo |
| V-Party | `V-Dem-CPD-Party-V2.rds` | `vparty/V-Dem-CPD-Party-V2.rds` | `data-setup/R/process_expert.R` | Not redistributed in this repo |
| POPPA | `poppa_integrated_v2.rds` | `poppa/poppa_integrated_v2.rds` | `data-setup/R/process_expert.R` | Not redistributed in this repo |
| Global Party Survey 2019 | `Global Party Survey by Party SPSS V2_1_Apr_2020-2.tab` | `gps/Global Party Survey by Party SPSS V2_1_Apr_2020-2.tab` | `data-setup/R/process_expert.R` | Not redistributed in this repo |
| Morgan historical expert data | `morgan_positions_raw.csv` | `morgan/morgan_positions_raw.csv` | `data-setup/R/process_morgan.R` | Not redistributed in this repo |
| PartyFacts crosswalk | `partyfacts-external-parties.csv` | `partyfacts/partyfacts-external-parties.csv` | source harmonization scripts | Not redistributed in this repo |

## PolDem download

Dataset: `poldem-election_all` from the PolDem Election Campaigns collection.

- Overview: <https://poldem.eui.eu/data-overview/>
- Download page: <https://poldem.eui.eu/download/election-campaigns/>
- CSV URL used locally on 2026-06-11:
  <https://poldem.eui.eu/downloads/cosa/poldem-election_all.csv>

Observed checksum after download on 2026-06-11:

```text
sha256  2cd8c9108b1b0b9c1b6594bb21acee709c70259cd02f450bc69fc09b505fc9fb
```

## Preflight check

From the repository root, run:

```bash
bash data-setup/run_data_setup.sh --dry-run
```

To check local raw file placement and print byte sizes/checksums, run:

```bash
bash data-setup/check_raw_data.sh
```

After restricted/manual files are in place, run the full local rebuild and comparison test with:

```bash
bash data-setup/run_data_setup.sh --full-test
```

The generated files remain under `_local/generated-inputs/`. They are compared to
the committed model-ready inputs, but never copied over them automatically.

## Processed files kept in this repository

The committed files under `data/` are limited to the model-ready inputs used by
the Julia/Stan estimation path:

- `data/text_data.csv`
- `data/expert.csv`
- `data/lr_data.csv`
- `data/union_mapping.csv`
- `data/party_families.csv`

These files document the analysis-ready inputs while avoiding redistribution of
the underlying raw source data.

See `data-setup/source_manifest.csv` for a machine-readable source checklist.

## Scripted download status

`data-setup/download_sources.py` downloads all sources that are script-accessible
without project-specific credentials: PolDem, PartyFacts, CHES family files,
POPPA from Harvard Dataverse, GPS from Harvard Dataverse, and V-Party through the
provider form when `PARTY2D_VDEM_EMAIL` is set.

The Manifesto Project main dataset requires a Manifesto API key/login. Set
`MANIFESTO_API_KEY` or `PARTY2D_MANIFESTO_API_KEY` for scripted download.

The Morgan historical file is a local OCR/transcription source from Morgan
(1976), not a public provider download. It can be provided on request and must be
placed locally at `$PARTY2D_RAW_DATA_DIR/morgan/morgan_positions_raw.csv` for
full rebuild tests.