# Raw data sources and local-only setup

This repository includes model-ready inputs under `data/`, but it does not
redistribute licensed, restricted, or third-party raw source files. Raw files
needed to regenerate processed inputs should be kept in `_local/raw/` or another
directory selected with `PARTY2D_RAW_DATA_DIR`.

The source setup scripts document how to obtain and test those files locally.
Users must use their own Manifesto Project API key and obtain the Morgan OCR/
transcription file separately.

The data setup workflow writes intermediates to `_local/build/`, regenerated
inputs to `_local/generated-inputs/`, and comparison reports to `_local/reports/`.
It does not overwrite committed `data/` files.

Recommended local layout:

```text
_local/raw/
  poldem/poldem-election_all.csv
  manifesto/MPDataset_MPDS2025a.csv
  partyfacts/partyfacts-external-parties.csv
  ches/...
  vparty/...
  poppa/...
  gps/...
  morgan/...
```

The scripts read `PARTY2D_RAW_DATA_DIR` when it is set. For another location, use:

```bash
export PARTY2D_RAW_DATA_DIR=/path/to/local/raw
```

## Required raw inputs

| Source | Raw file | Local path below `$PARTY2D_RAW_DATA_DIR` | Used by | Redistribution |
| --- | --- | --- | --- | --- |
| Manifesto Project Dataset | `MPDataset_MPDS2025a.csv` | `manifesto/MPDataset_MPDS2025a.csv` | `data-setup/R/process_manifesto.R` | Not redistributed in this repo |
| PolDem Election Campaigns, all issues | `poldem-election_all.csv` | `poldem/poldem-election_all.csv` | `data-setup/R/process_poldem.R` | Not redistributed in this repo |
| CHES family | CHES aggregate and expert-level files | `ches/` | `data-setup/R/process_expert.R` | Not redistributed in this repo |
| V-Party | `V-Dem-CPD-Party-V2.rds` | `vparty/V-Dem-CPD-Party-V2.rds` | `data-setup/R/process_expert.R` | Not redistributed in this repo |
| POPPA | `poppa_integrated_v2.rds` | `poppa/poppa_integrated_v2.rds` | `data-setup/R/process_expert.R` | Not redistributed in this repo |
| Global Party Survey 2019 | `Global Party Survey by Party SPSS V2_1_Apr_2020-2.tab` | `gps/Global Party Survey by Party SPSS V2_1_Apr_2020-2.tab` | `data-setup/R/process_expert.R` | Not redistributed in this repo |
| Morgan historical expert data | `morgan_positions_raw.csv` | `morgan/morgan_positions_raw.csv` | `data-setup/R/process_morgan.R` | Not redistributed in this repo |
| PartyFacts crosswalk | `partyfacts-external-parties.csv` | `partyfacts/partyfacts-external-parties.csv` | source harmonization scripts | Not redistributed in this repo |

## PolDem download

Dataset: `poldem-election_all` from the PolDem Election Campaigns collection.

- Overview: <https://poldem.eui.eu/data-overview/>
- Download page: <https://poldem.eui.eu/download/election-campaigns/>
- CSV URL used locally on 2026-06-11:
  <https://poldem.eui.eu/downloads/cosa/poldem-election_all.csv>

Observed checksum after download on 2026-06-11:

```text
sha256  2cd8c9108b1b0b9c1b6594bb21acee709c70259cd02f450bc69fc09b505fc9fb
```

## Preflight check

From the repository root, run:

```bash
bash data-setup/run_data_setup.sh --dry-run
```

To check local raw file placement and print byte sizes/checksums, run:

```bash
bash data-setup/check_raw_data.sh
```

After restricted/manual files are in place, run the full local rebuild and comparison test with:

```bash
bash data-setup/run_data_setup.sh
```

The generated files remain under `_local/generated-inputs/`. They are compared to
the committed model-ready inputs, but never copied over them automatically.

## Processed files kept in this repository

The committed files under `data/` are limited to the model-ready inputs used by
the Julia/Stan estimation path:

- `data/text_data.csv`
- `data/expert.csv`
- `data/lr_data.csv`
- `data/union_mapping.csv`
- `data/party_families.csv`

These files document the analysis-ready inputs while avoiding redistribution of
the underlying raw source data.

See `data-setup/source_manifest.csv` for a machine-readable source checklist.

## Source access status

The public repository does not include an automatic source downloader. Obtain
source files directly from the providers under their terms and place them in the
documented local raw-data layout before running the rebuild checks.

The Manifesto Project main dataset requires Manifesto Project access/API
credentials from the provider.

The Morgan historical file is a local OCR/transcription source from Morgan
(1976), not a public provider download. It can be provided on request and must be
placed locally at `$PARTY2D_RAW_DATA_DIR/morgan/morgan_positions_raw.csv` for
full rebuild tests.