aseimel
68 lines
2.1 KiB
Markdown
68 lines
2.1 KiB
Markdown
# Raw data sources and local-only setup
|
|
|
|
The public replication repository includes processed inputs under `data/`, but it
|
|
does not redistribute licensed or third-party raw source files. Raw files needed
|
|
to regenerate processed inputs should be kept in a local-only directory outside
|
|
the replication git repository.
|
|
|
|
Recommended local layout:
|
|
|
|
```text
|
|
_local/raw/
|
|
poldem/poldem-election_all.csv
|
|
manifesto/MPDataset_MPDS2025a.csv
|
|
```
|
|
|
|
The scripts read `PARTY2D_RAW_DATA_DIR` when it is set. For another location, use:
|
|
|
|
```bash
|
|
export PARTY2D_RAW_DATA_DIR=/path/to/local/raw
|
|
```
|
|
|
|
## Required raw inputs
|
|
|
|
| Source | Raw file | Local path below `$PARTY2D_RAW_DATA_DIR` | Used by | Redistribution |
|
|
| --- | --- | --- | --- | --- |
|
|
| PolDem Election Campaigns, all issues | `poldem-election_all.csv` | `poldem/poldem-election_all.csv` | `src/r/00c_process_poldem.R` | Not redistributed in this repo |
|
|
| Manifesto Project Dataset | `MPDataset_MPDS2025a.csv` | `manifesto/MPDataset_MPDS2025a.csv` | `src/r/00a_process_manifesto.R` if `data/manifesto_data.csv` is regenerated | Not redistributed in this repo |
|
|
|
|
## PolDem download
|
|
|
|
Dataset: `poldem-election_all` from the PolDem Election Campaigns collection.
|
|
|
|
- Overview: <https://poldem.eui.eu/data-overview/>
|
|
- Download page: <https://poldem.eui.eu/download/election-campaigns/>
|
|
- CSV URL used locally on 2026-06-11:
|
|
<https://poldem.eui.eu/downloads/cosa/poldem-election_all.csv>
|
|
|
|
Observed checksum after download on 2026-06-11:
|
|
|
|
```text
|
|
sha256 2cd8c9108b1b0b9c1b6594bb21acee709c70259cd02f450bc69fc09b505fc9fb
|
|
```
|
|
|
|
## Preflight check
|
|
|
|
From the repository root, run:
|
|
|
|
```bash
|
|
bash src/sh/check_raw_data.sh
|
|
```
|
|
|
|
This checks required local raw inputs and prints byte sizes and SHA-256 checksums.
|
|
|
|
## Processed files kept in this repository
|
|
|
|
The processed files under `data/` are intended to be part of this repository,
|
|
including:
|
|
|
|
- `data/poldem_data.csv`
|
|
- `data/manifesto_data.csv`
|
|
- `data/text_data.csv`
|
|
- `data/expert.csv`
|
|
- `data/lr_data.csv`
|
|
- `data/model_data.csv`
|
|
|
|
These files document the analysis-ready inputs while avoiding redistribution of
|
|
the underlying raw source data.
|