# party2d

Code and processed model inputs for generating two-dimensional party-position estimates from text and expert data. The model combines manifesto and media text indicators with expert survey placements in a Bayesian dynamic item-response framework.

## Repository contents

- `data-setup/` — source download, source-file checks, and rebuild workflow for raw files that cannot be redistributed here.
- `src/julia/` — Stan data preparation, model fitting, post-estimation, enrichment, and validation.
- `models/` — Stan model specification.
- `data/` — processed party-level inputs used by the Julia/Stan model.
- `metadata/` — data dictionary and source-support documentation.
- `diagnostics/` — repository diagnostics report regenerated after model estimation.
- `data/releases/` — release-ready data files, checksums, and diagnostics report.

Processed inputs needed by the model are included in `data/` so the estimation step can be reproduced from the model-ready data.

## Source data and redistribution

The public repository does **not** contain the original raw/source files. Several inputs are third-party datasets with their own terms of use, and the Morgan historical file is a local OCR/transcription source. Instead, this repository provides:

1. committed model-ready inputs in `data/`, sufficient for fitting the Julia/Stan model; and
2. `data-setup/`, an R/Shell workflow that downloads script-accessible sources, checks locally supplied raw sources, and rebuilds comparable model-ready inputs locally.

`data-setup/` downloads script-accessible sources and checks locally supplied sources for:

- PolDem
- PartyFacts crosswalk
- CHES family files
- POPPA from Harvard Dataverse
- Global Party Survey 2019 from Harvard Dataverse
- V-Party through the provider's download form

Two inputs require user-provided access/material:

- **Manifesto Project**: users must obtain the source data through their own Manifesto Project access.
- **Morgan historical expert data**: `morgan_positions_raw.csv` is not publicly downloadable; it can be provided on request and should be placed locally under `_local/raw/morgan/`.

The setup workflow never overwrites committed files in `data/`. See `data-setup/README.md` for exact commands and source details.

Run the full source-data setup workflow with:

```bash
bash data-setup/run_data_setup.sh
```

This downloads script-accessible source files, checks required local files, rebuilds model-ready inputs locally, and writes a comparison report. Manifesto Project requires your own provider access, and the Morgan OCR/transcription file can be provided on request.

## Running the pipeline

Run the full workflow with:

```bash
bash run_estimation.sh full
```

This checks that model-ready inputs are present, then executes the Julia/Stan workflow scripts:

```bash
bash scripts/01_prepare_data.sh   # checks model-ready inputs; does not rebuild raw data
bash scripts/02_fit_model.sh
bash scripts/03_extract_estimates.sh
bash scripts/04_enrich_estimates.sh
bash scripts/05_validate_estimates.sh
```

The numbered scripts can also be run manually in that order.

The Bayesian model is computationally expensive. The production run used 4 cores on an AMD Ryzen 9 7945HX and took 60,372 seconds, approximately 16 hours 46 minutes.

If model output is already available, rebuild estimates without refitting Stan:

```bash
bash run_estimation.sh reuse
```

`reuse` verifies the model-ready inputs, then reruns post-estimation, enrichment, and validation while skipping the Stan fitting step.

To check the local setup without fitting the model, run:

```bash
bash run_estimation.sh dry-run
```

After estimation and validation have been run, regenerate the diagnostics report with:

```bash
Rscript diagnostics/generate_diagnostics.R
```

The report is written to `diagnostics/generated/diagnostics_report.pdf` and copied to `data/releases/party_2d_diagnostics_report_v0.pdf`.

## Data inputs

The model-ready inputs are included under `data/`:

- `text_data.csv`
- `expert.csv`
- `lr_data.csv`
- `union_mapping.csv`
- `party_families.csv`

Original raw source files are not redistributed. Rebuilding inputs from raw files is separate from the normal estimation workflow and never replaces committed `data/` inputs automatically.

## Output dimensions

The two position dimensions are scaled from 0 to 1:

- Economic left-right: economic left to economic right.
- Cultural cosmopolitan--traditionalist: cosmopolitan to traditionalist.

Column definitions are in `metadata/data_dictionary.csv`.

## Release files

The public release assets are:

- `party_2d_election_year_panel_v0.zip`
- `party_2d_annual_model_output_v0.zip`
- `party_2d_diagnostics_report_v0.pdf`
- `SHA256SUMS`

`SHA256SUMS` hashes the release assets listed above.

## License

This release uses the Creative Commons Attribution 4.0 International License
(CC BY 4.0), the standard open license expected for Scientific Data data
descriptors. The license applies to release materials created for this
repository; third-party source data remain governed by their own terms. See
`LICENSE`.