# party2d

Code and processed model inputs for generating two-dimensional party-position estimates from text and expert data. The model combines manifesto and media text indicators with expert survey placements in a Bayesian dynamic item-response framework.

## Repository contents

- `data-setup/` — source download and rebuild workflow for raw files that cannot be redistributed here.
- `src/julia/` — Stan data preparation, model fitting, post-estimation, enrichment, and validation.
- `models/` — Stan model specification.
- `data/` — five processed party-level inputs used by the Julia/Stan model.
- `metadata/` — data dictionary and source-support documentation.
- `docs/` — raw data source documentation, coding decisions, and operational notes.
- `diagnostics/` — repository diagnostics report regenerated after model estimation.

Processed inputs needed by the model are included in `data/` so the estimation step can be reproduced from the model-ready data.

## Source data and redistribution

The public repository does **not** contain the original raw/source files. Several inputs are third-party datasets with their own terms of use, and the Morgan historical file is a local OCR/transcription source. Instead, this repository provides:

1. committed model-ready inputs in `data/`, sufficient for fitting the Julia/Stan model; and
2. `data-setup/`, a workflow that downloads or checks raw sources and rebuilds comparable model-ready inputs locally.

`data-setup/` automatically downloads all script-accessible sources:

- PolDem
- PartyFacts crosswalk
- CHES family files
- POPPA from Harvard Dataverse
- Global Party Survey 2019 from Harvard Dataverse
- V-Party through the provider's download form

Two inputs require user-provided access/material:

- **Manifesto Project**: users must provide their own Manifesto Project API key via `MANIFESTO_API_KEY` or `PARTY2D_MANIFESTO_API_KEY`.
- **Morgan historical expert data**: `morgan_positions_raw.csv` is not publicly downloadable; it can be provided on request and should be placed locally under `_local/raw/morgan/`.

The setup workflow never overwrites committed files in `data/`. See `data-setup/README.md` and `docs/RAW_DATA_SOURCES.md` for exact commands and source details.

Run the full source-data setup workflow with:

```bash
bash data-setup/run_data_setup.sh
```

This downloads script-accessible source files, checks required local files, rebuilds model-ready inputs locally, and writes a comparison report. Manifesto Project requires your own API key, and the Morgan OCR/transcription file can be provided on request.

## Running the pipeline

Run the full workflow with:

```bash
bash run_estimation.sh full
```

This checks that model-ready inputs are present, then executes the Julia/Stan workflow scripts:

```bash
bash scripts/01_prepare_data.sh   # checks model-ready inputs; does not rebuild raw data
bash scripts/02_fit_model.sh
bash scripts/03_extract_estimates.sh
bash scripts/04_enrich_estimates.sh
bash scripts/05_validate_estimates.sh
```

The numbered scripts can also be run manually in that order.

The Bayesian model is computationally expensive. The production run used 4 cores on an AMD Ryzen 9 7945HX and took 60,372 seconds, approximately 16 hours 46 minutes.

If model output is already available, rebuild estimates without refitting Stan:

```bash
bash run_estimation.sh reuse
```

`reuse` verifies the model-ready inputs, then reruns post-estimation, enrichment, and validation while skipping the Stan fitting step.

To check the local setup without fitting the model, run:

```bash
bash run_estimation.sh dry-run
```

After estimation and validation have been run, regenerate the diagnostics report with:

```bash
Rscript diagnostics/generate_diagnostics.R
```

The report is written to `diagnostics/generated/diagnostics_report.pdf` and copied to `data/releases/party_2d_diagnostics_report_v0.pdf`.

## Data inputs

The model-ready inputs are included under `data/`:

- `text_data.csv`
- `expert.csv`
- `lr_data.csv`
- `union_mapping.csv`
- `party_families.csv`

Original raw source files are not redistributed. Rebuilding inputs from raw files is separate from the normal estimation workflow and never replaces committed `data/` inputs automatically.

## Output dimensions

The two position dimensions are scaled from 0 to 1:

- Economic left-right: economic left to economic right.
- Cultural cosmopolitan--traditionalist: cosmopolitan to traditionalist.

Column definitions are in `metadata/data_dictionary.csv`.