# Raw data sources and local-only setup

The public replication repository includes processed inputs under `data/`, but it
does not redistribute licensed or third-party raw source files. Raw files needed
to regenerate processed inputs should be kept in a local-only directory outside
the replication git repository.

Recommended local layout:

```text
_local/raw/
  poldem/poldem-election_all.csv
  manifesto/MPDataset_MPDS2025a.csv
```

The scripts read `PARTY2D_RAW_DATA_DIR` when it is set. For another location, use:

```bash
export PARTY2D_RAW_DATA_DIR=/path/to/local/raw
```

## Required raw inputs

| Source | Raw file | Local path below `$PARTY2D_RAW_DATA_DIR` | Used by | Redistribution |
| --- | --- | --- | --- | --- |
| PolDem Election Campaigns, all issues | `poldem-election_all.csv` | `poldem/poldem-election_all.csv` | `src/r/00c_process_poldem.R` | Not redistributed in this repo |
| Manifesto Project Dataset | `MPDataset_MPDS2025a.csv` | `manifesto/MPDataset_MPDS2025a.csv` | `src/r/00a_process_manifesto.R` if `data/manifesto_data.csv` is regenerated | Not redistributed in this repo |

## PolDem download

Dataset: `poldem-election_all` from the PolDem Election Campaigns collection.

- Overview: <https://poldem.eui.eu/data-overview/>
- Download page: <https://poldem.eui.eu/download/election-campaigns/>
- CSV URL used locally on 2026-06-11:
  <https://poldem.eui.eu/downloads/cosa/poldem-election_all.csv>

Observed checksum after download on 2026-06-11:

```text
sha256  2cd8c9108b1b0b9c1b6594bb21acee709c70259cd02f450bc69fc09b505fc9fb
```

## Preflight check

From the repository root, run:

```bash
bash src/sh/check_raw_data.sh
```

This checks required local raw inputs and prints byte sizes and SHA-256 checksums.

## Processed files kept in this repository

The processed files under `data/` are intended to be part of this repository,
including:

- `data/poldem_data.csv`
- `data/manifesto_data.csv`
- `data/text_data.csv`
- `data/expert.csv`
- `data/lr_data.csv`
- `data/model_data.csv`

These files document the analysis-ready inputs while avoiding redistribution of
the underlying raw source data.