Clarify source data setup requirements

README.md

@@ -4,7 +4,7 @@ Code and processed model inputs for generating two-dimensional party-position es
 
 ## Repository contents
 
-- `data-setup/` — optional scripts and source manifest for rebuilding model-ready inputs from local raw files.
+- `data-setup/` — local-only source download and rebuild workflow. It is included because the original source files cannot be redistributed in this public repository.
 - `src/julia/` — Stan data preparation, model fitting, post-estimation, enrichment, and validation.
 - `models/` — Stan model specification.
 - `data/` — five processed party-level inputs used by the Julia/Stan model.
@@ -13,6 +13,29 @@ Code and processed model inputs for generating two-dimensional party-position es
 
 Processed inputs needed by the model are included in `data/` so the estimation step can be reproduced from the model-ready data.
 
+## Source data and redistribution
+
+The public repository intentionally does **not** contain the original raw/source files. Several inputs are third-party datasets with their own terms of use, and the Morgan historical file is a local OCR/transcription source. Instead, the repository provides:
+
+1. committed model-ready inputs in `data/`, sufficient for fitting the Julia/Stan model; and
+2. `data-setup/`, a local-only workflow that downloads or checks raw sources and rebuilds comparable model-ready inputs under ignored `_local/` directories.
+
+`data-setup/` automatically downloads all script-accessible sources:
+
+- PolDem
+- PartyFacts crosswalk
+- CHES family files
+- POPPA from Harvard Dataverse
+- Global Party Survey 2019 from Harvard Dataverse
+- V-Party through the provider's download form
+
+Two inputs require user-provided access/material:
+
+- **Manifesto Project**: users must provide their own Manifesto Project API key via `MANIFESTO_API_KEY` or `PARTY2D_MANIFESTO_API_KEY`.
+- **Morgan historical expert data**: `morgan_positions_raw.csv` is not publicly downloadable; it can be provided on request and should be placed locally under `_local/raw/morgan/`.
+
+Raw downloads, intermediate build files, regenerated inputs, R package installs, and comparison reports are written only under `_local/`, which is ignored by git. The setup workflow never overwrites committed files in `data/`. See `data-setup/README.md` and `docs/RAW_DATA_SOURCES.md` for exact commands and source details.
+
 ## Running the pipeline
 
 Run the full workflow with:
@@ -59,7 +82,7 @@ The model-ready inputs are included under `data/`:
 - `union_mapping.csv`
 - `party_families.csv`
 
-Original raw source files are not redistributed. See `docs/RAW_DATA_SOURCES.md` and `data-setup/` for the list of original data sources, access information, and expected local filenames for regenerating the processed inputs. Rebuilding inputs from raw files is separate from the normal estimation workflow and writes only to ignored `_local/` test directories; it never replaces committed `data/` inputs automatically.
+Original raw source files are not redistributed. Rebuilding inputs from raw files is separate from the normal estimation workflow and writes only to ignored `_local/` directories; it never replaces committed `data/` inputs automatically.
 
 ## Output variables
 

data-setup/README.md

@@ -1,11 +1,30 @@
 # Data setup
 
-This directory documents and scripts the optional process for rebuilding the model-ready inputs from local raw/source datasets. The main estimation workflow does not run these scripts; once the five model-ready CSVs exist in `data/`, fitting and post-estimation are Julia/Stan-only.
+This directory exists because the public repository cannot redistribute the original raw/source files. It provides a reproducible, local-only workflow to download the source files that can be downloaded automatically, check user-provided restricted files, rebuild model-ready inputs, and compare the rebuilt inputs with the committed files in `../data/`.
+
+The main estimation workflow does not run these scripts. Once the five model-ready CSVs exist in `data/`, fitting and post-estimation are Julia/Stan-only.
 
 The setup workflow is intentionally local-only. It never overwrites committed files in `data/`. Raw downloads, intermediate files, regenerated inputs, and comparison reports go under `_local/`, which is ignored by git.
 
 Raw source files are not redistributed in this repository. Put them in `_local/raw/` or set `PARTY2D_RAW_DATA_DIR` to another local directory. See `source_manifest.csv` and `../docs/RAW_DATA_SOURCES.md` for source-specific access notes.
 
+## What downloads automatically?
+
+`data-setup/download_sources.py` automatically downloads all source files that are script-accessible under the providers' terms:
+
+| Source | Automatic? | Requirement |
+| --- | --- | --- |
+| PolDem | Yes | none |
+| PartyFacts crosswalk | Yes | none |
+| CHES family files | Yes | none |
+| POPPA | Yes | none; downloaded from Harvard Dataverse |
+| Global Party Survey 2019 | Yes | none; downloaded from Harvard Dataverse |
+| V-Party | Yes | set `PARTY2D_VDEM_EMAIL`; optionally set `PARTY2D_VDEM_GENDER` |
+| Manifesto Project | Yes, with credentials | set your own `MANIFESTO_API_KEY` or `PARTY2D_MANIFESTO_API_KEY` |
+| Morgan historical expert data | No | place `morgan_positions_raw.csv` locally; available on request |
+
+Do not commit any downloaded or user-provided source files. All default paths are under ignored `_local/` directories.
+
 Recommended local layout:
 
 ```text
@@ -28,29 +47,28 @@ _local/generated-inputs/  # regenerated final model-input CSVs
 _local/reports/           # comparison reports
 ```
 
+## Commands
+
 Check setup without downloading or rebuilding:
 
 ```bash
 bash data-setup/run_data_setup.sh --dry-run
 ```
 
-Attempt automatic downloads where permitted/available:
+Download all automatically downloadable sources. This also downloads Manifesto if you set a Manifesto API key and V-Party if you set a V-Dem form email:
 
 ```bash
 bash data-setup/run_data_setup.sh --download-only
 ```
 
-The downloader handles public direct downloads, Harvard Dataverse files, and the
-V-Dem form workflow. For V-Dem, set an email address accepted by the provider's
-download form:
+For V-Party, set an email address accepted by the provider's download form:
 
 ```bash
 export PARTY2D_VDEM_EMAIL='you@example.org'
 export PARTY2D_VDEM_GENDER=''   # blank means prefer not to say
 ```
 
-The Manifesto Project main dataset is behind the Manifesto API/login terms. To
-download it through the script, set one of:
+For Manifesto Project, users must use their own API key from their Manifesto Project account:
 
 ```bash
 export MANIFESTO_API_KEY='...'
@@ -58,9 +76,11 @@ export MANIFESTO_API_KEY='...'
 export PARTY2D_MANIFESTO_API_KEY='...'
 ```
 
-`morgan/morgan_positions_raw.csv` is a local OCR/transcription source derived
-from Morgan (1976), not a public provider download. Place it under
-`_local/raw/morgan/` before a full rebuild test.
+`morgan/morgan_positions_raw.csv` is a local OCR/transcription source derived from Morgan (1976), not a public provider download. It can be provided on request. Place it under `_local/raw/morgan/` before a full rebuild test:
+
+```text
+_local/raw/morgan/morgan_positions_raw.csv
+```
 
 Rebuild model-ready inputs after placing all required local files. This writes to `_local/generated-inputs/`, not `data/`:
 
@@ -74,10 +94,12 @@ Compare regenerated inputs with the committed inputs:
 bash data-setup/run_data_setup.sh --compare
 ```
 
-Run the full local test sequence — public downloads where available, raw-file preflight, local rebuild, and comparison — after manually placing restricted sources:
+Run the full local test sequence — source downloads, raw-file preflight, local rebuild, and comparison — after setting required environment variables and placing Morgan locally:
 
 ```bash
 bash data-setup/run_data_setup.sh --full-test
 ```
 
-The comparison writes `_local/reports/input_comparison.md` and exits nonzero if any generated input is missing or differs. Replacing committed inputs, if ever needed, is a separate manual decision and is not done by these scripts.
+The comparison writes `_local/reports/input_comparison.md`. Replacing committed inputs, if ever needed, is a separate manual decision and is not done by these scripts.
+
+Known behavior: the current public/rebuilt sources run through the workflow successfully, but regenerated `text_data.csv`, `expert.csv`, and `lr_data.csv` are not byte-identical to the committed model-ready inputs because of source-version and linkage differences. The comparison report records those differences explicitly.

docs/RAW_DATA_SOURCES.md

@@ -5,6 +5,11 @@ redistribute licensed, restricted, or third-party raw source files. Raw files
 needed to regenerate processed inputs should be kept in `_local/raw/` or another
 local-only directory selected with `PARTY2D_RAW_DATA_DIR`.
 
+The source setup scripts are provided specifically to avoid redistributing those
+source files while still documenting how to obtain and test them locally. Users
+must use their own Manifesto Project API key and must obtain the Morgan OCR/
+transcription file separately.
+
 The data setup workflow writes intermediates to `_local/build/`, regenerated
 inputs to `_local/generated-inputs/`, and comparison reports to `_local/reports/`.
 It does not overwrite committed `data/` files.
@@ -108,5 +113,6 @@ The Manifesto Project main dataset requires a Manifesto API key/login. Set
 `MANIFESTO_API_KEY` or `PARTY2D_MANIFESTO_API_KEY` for scripted download.
 
 The Morgan historical file is a local OCR/transcription source from Morgan
-(1976), not a public provider download. It must be placed locally at
-`$PARTY2D_RAW_DATA_DIR/morgan/morgan_positions_raw.csv` for full rebuild tests.
+(1976), not a public provider download. It can be provided on request and must be
+placed locally at `$PARTY2D_RAW_DATA_DIR/morgan/morgan_positions_raw.csv` for
+full rebuild tests.