armin/party2d
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Tighten source setup documentation

Browse Source
This commit is contained in:
aseimel
2026-06-15 14:53:20 +02:00
parent 8353d02761
commit 241e9e8a8d
3 changed files with 17 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+5 -5
View File
@@ -4,7 +4,7 @@ Code and processed model inputs for generating two-dimensional party-position es
## Repository contents
- `data-setup/` — local-only source download and rebuild workflow. It is included because the original source files cannot be redistributed in this public repository.
- `data-setup/` — source download and rebuild workflow for raw files that cannot be redistributed here.
- `src/julia/` — Stan data preparation, model fitting, post-estimation, enrichment, and validation.
- `models/` — Stan model specification.
- `data/` — five processed party-level inputs used by the Julia/Stan model.
@@ -15,10 +15,10 @@ Processed inputs needed by the model are included in `data/` so the estimation s
## Source data and redistribution
The public repository intentionally does **not** contain the original raw/source files. Several inputs are third-party datasets with their own terms of use, and the Morgan historical file is a local OCR/transcription source. Instead, the repository provides:
The public repository does **not** contain the original raw/source files. Several inputs are third-party datasets with their own terms of use, and the Morgan historical file is a local OCR/transcription source. Instead, this repository provides:
1. committed model-ready inputs in `data/`, sufficient for fitting the Julia/Stan model; and
2. `data-setup/`, a local-only workflow that downloads or checks raw sources and rebuilds comparable model-ready inputs under ignored `_local/` directories.
2. `data-setup/`, a workflow that downloads or checks raw sources and rebuilds comparable model-ready inputs locally.
`data-setup/` automatically downloads all script-accessible sources:
@@ -34,7 +34,7 @@ Two inputs require user-provided access/material:
- **Manifesto Project**: users must provide their own Manifesto Project API key via `MANIFESTO_API_KEY` or `PARTY2D_MANIFESTO_API_KEY`.
- **Morgan historical expert data**: `morgan_positions_raw.csv` is not publicly downloadable; it can be provided on request and should be placed locally under `_local/raw/morgan/`.
Raw downloads, intermediate build files, regenerated inputs, R package installs, and comparison reports are written only under `_local/`, which is ignored by git. The setup workflow never overwrites committed files in `data/`. See `data-setup/README.md` and `docs/RAW_DATA_SOURCES.md` for exact commands and source details.
The setup workflow never overwrites committed files in `data/`. See `data-setup/README.md` and `docs/RAW_DATA_SOURCES.md` for exact commands and source details.
## Running the pipeline
@@ -82,7 +82,7 @@ The model-ready inputs are included under `data/`:
- `union_mapping.csv`
- `party_families.csv`
Original raw source files are not redistributed. Rebuilding inputs from raw files is separate from the normal estimation workflow and writes only to ignored `_local/` directories; it never replaces committed `data/` inputs automatically.
Original raw source files are not redistributed. Rebuilding inputs from raw files is separate from the normal estimation workflow and never replaces committed `data/` inputs automatically.
## Output variables
data-setup/README.md
+9 -9
View File
@@ -1,12 +1,12 @@
# Data setup
This directory exists because the public repository cannot redistribute the original raw/source files. It provides a reproducible, local-only workflow to download the source files that can be downloaded automatically, check user-provided restricted files, rebuild model-ready inputs, and compare the rebuilt inputs with the committed files in `../data/`.
This directory exists because the public repository cannot redistribute the original raw/source files. It downloads the sources that can be fetched automatically, checks user-provided restricted files, rebuilds model-ready inputs, and compares them with the committed files in `../data/`.
The main estimation workflow does not run these scripts. Once the five model-ready CSVs exist in `data/`, fitting and post-estimation are Julia/Stan-only.
The setup workflow is intentionally local-only. It never overwrites committed files in `data/`. Raw downloads, intermediate files, regenerated inputs, and comparison reports go under `_local/`, which is ignored by git.
The setup workflow never overwrites committed files in `data/`.
Raw source files are not redistributed in this repository. Put them in `_local/raw/` or set `PARTY2D_RAW_DATA_DIR` to another local directory. See `source_manifest.csv` and `../docs/RAW_DATA_SOURCES.md` for source-specific access notes.
Put raw source files in `_local/raw/` or set `PARTY2D_RAW_DATA_DIR` to another local directory. See `source_manifest.csv` and `../docs/RAW_DATA_SOURCES.md` for source-specific access notes.
## What downloads automatically?
@@ -23,7 +23,7 @@ Raw source files are not redistributed in this repository. Put them in `_local/r
| Manifesto Project | Yes, with credentials | set your own `MANIFESTO_API_KEY` or `PARTY2D_MANIFESTO_API_KEY` |
| Morgan historical expert data | No | place `morgan_positions_raw.csv` locally; available on request |
Do not commit any downloaded or user-provided source files. All default paths are under ignored `_local/` directories.
Do not commit downloaded or user-provided source files.
Recommended local layout:
@@ -55,20 +55,20 @@ Check setup without downloading or rebuilding:
bash data-setup/run_data_setup.sh --dry-run
```
Download all automatically downloadable sources. This also downloads Manifesto if you set a Manifesto API key and V-Party if you set a V-Dem form email:
Download all script-accessible sources. Manifesto requires an API key; V-Party requires an email for the provider form:
```bash
bash data-setup/run_data_setup.sh --download-only
```
For V-Party, set an email address accepted by the provider's download form:
V-Party:
```bash
export PARTY2D_VDEM_EMAIL='you@example.org'
export PARTY2D_VDEM_GENDER='' # blank means prefer not to say
```
For Manifesto Project, users must use their own API key from their Manifesto Project account:
Manifesto Project:
```bash
export MANIFESTO_API_KEY='...'
@@ -76,13 +76,13 @@ export MANIFESTO_API_KEY='...'
export PARTY2D_MANIFESTO_API_KEY='...'
```
`morgan/morgan_positions_raw.csv` is a local OCR/transcription source derived from Morgan (1976), not a public provider download. It can be provided on request. Place it under `_local/raw/morgan/` before a full rebuild test:
Morgan is not a public provider download. The local OCR/transcription file can be provided on request and should be placed at:
```text
_local/raw/morgan/morgan_positions_raw.csv
```
Rebuild model-ready inputs after placing all required local files. This writes to `_local/generated-inputs/`, not `data/`:
Rebuild model-ready inputs after placing all required local files:
```bash
bash data-setup/run_data_setup.sh --build-test
docs/RAW_DATA_SOURCES.md
+3 -4
View File
@@ -3,11 +3,10 @@
This repository includes model-ready inputs under `data/`, but it does not
redistribute licensed, restricted, or third-party raw source files. Raw files
needed to regenerate processed inputs should be kept in `_local/raw/` or another
local-only directory selected with `PARTY2D_RAW_DATA_DIR`.
directory selected with `PARTY2D_RAW_DATA_DIR`.
The source setup scripts are provided specifically to avoid redistributing those
source files while still documenting how to obtain and test them locally. Users
must use their own Manifesto Project API key and must obtain the Morgan OCR/
The source setup scripts document how to obtain and test those files locally.
Users must use their own Manifesto Project API key and obtain the Morgan OCR/
transcription file separately.
The data setup workflow writes intermediates to `_local/build/`, regenerated