From 241e9e8a8d42157e29c7e933814c95d6ac72f397 Mon Sep 17 00:00:00 2001 From: aseimel Date: Mon, 15 Jun 2026 14:53:20 +0200 Subject: [PATCH] Tighten source setup documentation --- README.md | 10 +++++----- data-setup/README.md | 18 +++++++++--------- docs/RAW_DATA_SOURCES.md | 7 +++---- 3 files changed, 17 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index b753924..689e75e 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Code and processed model inputs for generating two-dimensional party-position es ## Repository contents -- `data-setup/` — local-only source download and rebuild workflow. It is included because the original source files cannot be redistributed in this public repository. +- `data-setup/` — source download and rebuild workflow for raw files that cannot be redistributed here. - `src/julia/` — Stan data preparation, model fitting, post-estimation, enrichment, and validation. - `models/` — Stan model specification. - `data/` — five processed party-level inputs used by the Julia/Stan model. @@ -15,10 +15,10 @@ Processed inputs needed by the model are included in `data/` so the estimation s ## Source data and redistribution -The public repository intentionally does **not** contain the original raw/source files. Several inputs are third-party datasets with their own terms of use, and the Morgan historical file is a local OCR/transcription source. Instead, the repository provides: +The public repository does **not** contain the original raw/source files. Several inputs are third-party datasets with their own terms of use, and the Morgan historical file is a local OCR/transcription source. Instead, this repository provides: 1. committed model-ready inputs in `data/`, sufficient for fitting the Julia/Stan model; and -2. `data-setup/`, a local-only workflow that downloads or checks raw sources and rebuilds comparable model-ready inputs under ignored `_local/` directories. +2. `data-setup/`, a workflow that downloads or checks raw sources and rebuilds comparable model-ready inputs locally. `data-setup/` automatically downloads all script-accessible sources: @@ -34,7 +34,7 @@ Two inputs require user-provided access/material: - **Manifesto Project**: users must provide their own Manifesto Project API key via `MANIFESTO_API_KEY` or `PARTY2D_MANIFESTO_API_KEY`. - **Morgan historical expert data**: `morgan_positions_raw.csv` is not publicly downloadable; it can be provided on request and should be placed locally under `_local/raw/morgan/`. -Raw downloads, intermediate build files, regenerated inputs, R package installs, and comparison reports are written only under `_local/`, which is ignored by git. The setup workflow never overwrites committed files in `data/`. See `data-setup/README.md` and `docs/RAW_DATA_SOURCES.md` for exact commands and source details. +The setup workflow never overwrites committed files in `data/`. See `data-setup/README.md` and `docs/RAW_DATA_SOURCES.md` for exact commands and source details. ## Running the pipeline @@ -82,7 +82,7 @@ The model-ready inputs are included under `data/`: - `union_mapping.csv` - `party_families.csv` -Original raw source files are not redistributed. Rebuilding inputs from raw files is separate from the normal estimation workflow and writes only to ignored `_local/` directories; it never replaces committed `data/` inputs automatically. +Original raw source files are not redistributed. Rebuilding inputs from raw files is separate from the normal estimation workflow and never replaces committed `data/` inputs automatically. ## Output variables diff --git a/data-setup/README.md b/data-setup/README.md index 99be5c9..4a77089 100644 --- a/data-setup/README.md +++ b/data-setup/README.md @@ -1,12 +1,12 @@ # Data setup -This directory exists because the public repository cannot redistribute the original raw/source files. It provides a reproducible, local-only workflow to download the source files that can be downloaded automatically, check user-provided restricted files, rebuild model-ready inputs, and compare the rebuilt inputs with the committed files in `../data/`. +This directory exists because the public repository cannot redistribute the original raw/source files. It downloads the sources that can be fetched automatically, checks user-provided restricted files, rebuilds model-ready inputs, and compares them with the committed files in `../data/`. The main estimation workflow does not run these scripts. Once the five model-ready CSVs exist in `data/`, fitting and post-estimation are Julia/Stan-only. -The setup workflow is intentionally local-only. It never overwrites committed files in `data/`. Raw downloads, intermediate files, regenerated inputs, and comparison reports go under `_local/`, which is ignored by git. +The setup workflow never overwrites committed files in `data/`. -Raw source files are not redistributed in this repository. Put them in `_local/raw/` or set `PARTY2D_RAW_DATA_DIR` to another local directory. See `source_manifest.csv` and `../docs/RAW_DATA_SOURCES.md` for source-specific access notes. +Put raw source files in `_local/raw/` or set `PARTY2D_RAW_DATA_DIR` to another local directory. See `source_manifest.csv` and `../docs/RAW_DATA_SOURCES.md` for source-specific access notes. ## What downloads automatically? @@ -23,7 +23,7 @@ Raw source files are not redistributed in this repository. Put them in `_local/r | Manifesto Project | Yes, with credentials | set your own `MANIFESTO_API_KEY` or `PARTY2D_MANIFESTO_API_KEY` | | Morgan historical expert data | No | place `morgan_positions_raw.csv` locally; available on request | -Do not commit any downloaded or user-provided source files. All default paths are under ignored `_local/` directories. +Do not commit downloaded or user-provided source files. Recommended local layout: @@ -55,20 +55,20 @@ Check setup without downloading or rebuilding: bash data-setup/run_data_setup.sh --dry-run ``` -Download all automatically downloadable sources. This also downloads Manifesto if you set a Manifesto API key and V-Party if you set a V-Dem form email: +Download all script-accessible sources. Manifesto requires an API key; V-Party requires an email for the provider form: ```bash bash data-setup/run_data_setup.sh --download-only ``` -For V-Party, set an email address accepted by the provider's download form: +V-Party: ```bash export PARTY2D_VDEM_EMAIL='you@example.org' export PARTY2D_VDEM_GENDER='' # blank means prefer not to say ``` -For Manifesto Project, users must use their own API key from their Manifesto Project account: +Manifesto Project: ```bash export MANIFESTO_API_KEY='...' @@ -76,13 +76,13 @@ export MANIFESTO_API_KEY='...' export PARTY2D_MANIFESTO_API_KEY='...' ``` -`morgan/morgan_positions_raw.csv` is a local OCR/transcription source derived from Morgan (1976), not a public provider download. It can be provided on request. Place it under `_local/raw/morgan/` before a full rebuild test: +Morgan is not a public provider download. The local OCR/transcription file can be provided on request and should be placed at: ```text _local/raw/morgan/morgan_positions_raw.csv ``` -Rebuild model-ready inputs after placing all required local files. This writes to `_local/generated-inputs/`, not `data/`: +Rebuild model-ready inputs after placing all required local files: ```bash bash data-setup/run_data_setup.sh --build-test diff --git a/docs/RAW_DATA_SOURCES.md b/docs/RAW_DATA_SOURCES.md index b372eac..d6ab534 100644 --- a/docs/RAW_DATA_SOURCES.md +++ b/docs/RAW_DATA_SOURCES.md @@ -3,11 +3,10 @@ This repository includes model-ready inputs under `data/`, but it does not redistribute licensed, restricted, or third-party raw source files. Raw files needed to regenerate processed inputs should be kept in `_local/raw/` or another -local-only directory selected with `PARTY2D_RAW_DATA_DIR`. +directory selected with `PARTY2D_RAW_DATA_DIR`. -The source setup scripts are provided specifically to avoid redistributing those -source files while still documenting how to obtain and test them locally. Users -must use their own Manifesto Project API key and must obtain the Morgan OCR/ +The source setup scripts document how to obtain and test those files locally. +Users must use their own Manifesto Project API key and obtain the Morgan OCR/ transcription file separately. The data setup workflow writes intermediates to `_local/build/`, regenerated