Document raw source file reference

2026-06-15 11:33:18 +02:00
commit b5ca9370f1
61 changed files with 293618 additions and 0 deletions
@@ -0,0 +1,67 @@
+# Raw data sources and local-only setup
+
+The public replication repository includes processed inputs under `data/`, but it
+does not redistribute licensed or third-party raw source files. Raw files needed
+to regenerate processed inputs should be kept in a local-only directory outside
+the replication git repository.
+
+Recommended local layout:
+
+```text
+_local/raw/
+  poldem/poldem-election_all.csv
+  manifesto/MPDataset_MPDS2025a.csv
+```
+
+The scripts read `PARTY2D_RAW_DATA_DIR` when it is set. For another location, use:
+
+```bash
+export PARTY2D_RAW_DATA_DIR=/path/to/local/raw
+```
+
+## Required raw inputs
+
+| Source | Raw file | Local path below `$PARTY2D_RAW_DATA_DIR` | Used by | Redistribution |
+| --- | --- | --- | --- | --- |
+| PolDem Election Campaigns, all issues | `poldem-election_all.csv` | `poldem/poldem-election_all.csv` | `src/r/00c_process_poldem.R` | Not redistributed in this repo |
+| Manifesto Project Dataset | `MPDataset_MPDS2025a.csv` | `manifesto/MPDataset_MPDS2025a.csv` | `src/r/00a_process_manifesto.R` if `data/manifesto_data.csv` is regenerated | Not redistributed in this repo |
+
+## PolDem download
+
+Dataset: `poldem-election_all` from the PolDem Election Campaigns collection.
+
+- Overview: <https://poldem.eui.eu/data-overview/>
+- Download page: <https://poldem.eui.eu/download/election-campaigns/>
+- CSV URL used locally on 2026-06-11:
+  <https://poldem.eui.eu/downloads/cosa/poldem-election_all.csv>
+
+Observed checksum after download on 2026-06-11:
+
+```text
+sha256  2cd8c9108b1b0b9c1b6594bb21acee709c70259cd02f450bc69fc09b505fc9fb
+```
+
+## Preflight check
+
+From the repository root, run:
+
+```bash
+bash src/sh/check_raw_data.sh
+```
+
+This checks required local raw inputs and prints byte sizes and SHA-256 checksums.
+
+## Processed files kept in this repository
+
+The processed files under `data/` are intended to be part of this repository,
+including:
+
+- `data/poldem_data.csv`
+- `data/manifesto_data.csv`
+- `data/text_data.csv`
+- `data/expert.csv`
+- `data/lr_data.csv`
+- `data/model_data.csv`
+
+These files document the analysis-ready inputs while avoiding redistribution of
+the underlying raw source data.