armin/party2d
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Restore source downloader in R

Browse Source
This commit is contained in:
Armin
2026-06-15 17:45:03 +02:00
parent dad75063fc
commit 9b925f340f
6 changed files with 265 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
data-setup/README.md
+36 -14
View File
@@ -1,6 +1,6 @@
# Data setup
This directory exists because the public repository cannot redistribute the original raw/source files. It checks user-provided source files, rebuilds model-ready inputs, and compares them with the committed files in `../data/`.
This directory exists because the public repository cannot redistribute the original raw/source files. It downloads script-accessible sources, checks user-provided source files, rebuilds model-ready inputs, and compares them with the committed files in `../data/`.
The main estimation workflow does not run these scripts. Once the five model-ready CSVs exist in `data/`, fitting and post-estimation are Julia/Stan-only.
@@ -8,20 +8,20 @@ The setup workflow never overwrites committed files in `data/`.
Put raw source files in `_local/raw/` or set `PARTY2D_RAW_DATA_DIR` to another local directory. See `source_manifest.csv` and `../docs/RAW_DATA_SOURCES.md` for source-specific access notes.
## Source availability
## What downloads automatically?
The public repository does not include an automatic source downloader. Users should obtain source files directly from the providers under the providers' terms and place them under `_local/raw/` or another directory selected with `PARTY2D_RAW_DATA_DIR`.
`data-setup/R/01_download_sources.R` downloads source files that are script-accessible under the providers' terms and reports sources that require credentials or manual local files.
| Source | Requirement |
| --- | --- |
| PolDem | obtain from provider |
| PartyFacts crosswalk | obtain from provider |
| CHES family files | obtain from provider |
| POPPA | obtain from Harvard Dataverse |
| Global Party Survey 2019 | obtain from Harvard Dataverse |
| V-Party | obtain from provider |
| Manifesto Project | use your own Manifesto Project access/API key |
| Morgan historical expert data | place `morgan_positions_raw.csv` locally; available on request |
| Source | Automatic? | Requirement |
| --- | --- | --- |
| PolDem | Yes | none |
| PartyFacts crosswalk | Yes | none |
| CHES family files | Yes | none where provider links are live |
| POPPA | Yes | none; downloaded from Harvard Dataverse |
| Global Party Survey 2019 | Yes | none; downloaded from Harvard Dataverse |
| V-Party | Yes, with provider form details | set `PARTY2D_VDEM_EMAIL`; optionally set `PARTY2D_VDEM_GENDER` |
| Manifesto Project | Yes, with credentials | set your own `MANIFESTO_API_KEY` or `PARTY2D_MANIFESTO_API_KEY` |
| Morgan historical expert data | No | place `morgan_positions_raw.csv` locally; available on request |
Do not commit downloaded or user-provided source files.
@@ -58,9 +58,10 @@ bash data-setup/run_data_setup.sh
Maintenance options:
- `--dry-run` — check setup scripts only.
- `--download-only` — download script-accessible sources.
- `--build-test` — rebuild model-ready inputs from existing local sources.
- `--compare` — compare regenerated inputs with committed `data/` inputs.
- no option — rebuild and compare locally from existing source files.
- no option — download, rebuild, and compare locally.
Minimal check:
@@ -68,6 +69,27 @@ Minimal check:
bash data-setup/run_data_setup.sh --dry-run
```
Download sources:
```bash
bash data-setup/run_data_setup.sh --download-only
```
V-Party:
```bash
export PARTY2D_VDEM_EMAIL='you@example.org'
export PARTY2D_VDEM_GENDER='' # blank means prefer not to say
```
Manifesto Project:
```bash
export MANIFESTO_API_KEY='...'
# or
export PARTY2D_MANIFESTO_API_KEY='...'
```
Morgan is not a public provider download. The local OCR/transcription file can be provided on request and should be placed at:
```text