armin/party2d
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Tighten source setup documentation

Browse Source
This commit is contained in:
aseimel
2026-06-15 14:53:20 +02:00
parent 8353d02761
commit 241e9e8a8d
3 changed files with 17 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
data-setup/README.md
+9 -9
View File
@@ -1,12 +1,12 @@
# Data setup
This directory exists because the public repository cannot redistribute the original raw/source files. It provides a reproducible, local-only workflow to download the source files that can be downloaded automatically, check user-provided restricted files, rebuild model-ready inputs, and compare the rebuilt inputs with the committed files in `../data/`.
This directory exists because the public repository cannot redistribute the original raw/source files. It downloads the sources that can be fetched automatically, checks user-provided restricted files, rebuilds model-ready inputs, and compares them with the committed files in `../data/`.
The main estimation workflow does not run these scripts. Once the five model-ready CSVs exist in `data/`, fitting and post-estimation are Julia/Stan-only.
The setup workflow is intentionally local-only. It never overwrites committed files in `data/`. Raw downloads, intermediate files, regenerated inputs, and comparison reports go under `_local/`, which is ignored by git.
The setup workflow never overwrites committed files in `data/`.
Raw source files are not redistributed in this repository. Put them in `_local/raw/` or set `PARTY2D_RAW_DATA_DIR` to another local directory. See `source_manifest.csv` and `../docs/RAW_DATA_SOURCES.md` for source-specific access notes.
Put raw source files in `_local/raw/` or set `PARTY2D_RAW_DATA_DIR` to another local directory. See `source_manifest.csv` and `../docs/RAW_DATA_SOURCES.md` for source-specific access notes.
## What downloads automatically?
@@ -23,7 +23,7 @@ Raw source files are not redistributed in this repository. Put them in `_local/r
| Manifesto Project | Yes, with credentials | set your own `MANIFESTO_API_KEY` or `PARTY2D_MANIFESTO_API_KEY` |
| Morgan historical expert data | No | place `morgan_positions_raw.csv` locally; available on request |
Do not commit any downloaded or user-provided source files. All default paths are under ignored `_local/` directories.
Do not commit downloaded or user-provided source files.
Recommended local layout:
@@ -55,20 +55,20 @@ Check setup without downloading or rebuilding:
bash data-setup/run_data_setup.sh --dry-run
```
Download all automatically downloadable sources. This also downloads Manifesto if you set a Manifesto API key and V-Party if you set a V-Dem form email:
Download all script-accessible sources. Manifesto requires an API key; V-Party requires an email for the provider form:
```bash
bash data-setup/run_data_setup.sh --download-only
```
For V-Party, set an email address accepted by the provider's download form:
V-Party:
```bash
export PARTY2D_VDEM_EMAIL='you@example.org'
export PARTY2D_VDEM_GENDER='' # blank means prefer not to say
```
For Manifesto Project, users must use their own API key from their Manifesto Project account:
Manifesto Project:
```bash
export MANIFESTO_API_KEY='...'
@@ -76,13 +76,13 @@ export MANIFESTO_API_KEY='...'
export PARTY2D_MANIFESTO_API_KEY='...'
```
`morgan/morgan_positions_raw.csv` is a local OCR/transcription source derived from Morgan (1976), not a public provider download. It can be provided on request. Place it under `_local/raw/morgan/` before a full rebuild test:
Morgan is not a public provider download. The local OCR/transcription file can be provided on request and should be placed at:
```text
_local/raw/morgan/morgan_positions_raw.csv
```
Rebuild model-ready inputs after placing all required local files. This writes to `_local/generated-inputs/`, not `data/`:
Rebuild model-ready inputs after placing all required local files:
```bash
bash data-setup/run_data_setup.sh --build-test