Wire local-only data setup workflow
This commit is contained in:
@@ -4,10 +4,10 @@ Code and processed model inputs for generating two-dimensional party-position es
|
||||
|
||||
## Repository contents
|
||||
|
||||
- `src/r/` — scripts that prepare processed model inputs from source datasets.
|
||||
- `data-setup/` — optional scripts and source manifest for rebuilding model-ready inputs from local raw files.
|
||||
- `src/julia/` — Stan data preparation, model fitting, post-estimation, enrichment, and validation.
|
||||
- `models/` — Stan model specification.
|
||||
- `data/` — processed party-level inputs used by the model.
|
||||
- `data/` — five processed party-level inputs used by the Julia/Stan model.
|
||||
- `metadata/` — data dictionary and source-support documentation.
|
||||
- `docs/` — raw data source documentation, coding decisions, and operational notes.
|
||||
|
||||
@@ -21,16 +21,17 @@ Run the full workflow with:
|
||||
bash run_estimation.sh full
|
||||
```
|
||||
|
||||
This executes the numbered workflow scripts:
|
||||
This checks that model-ready inputs are present, then executes the Julia/Stan workflow scripts:
|
||||
|
||||
```bash
|
||||
bash scripts/01_prepare_data.sh
|
||||
bash scripts/01_prepare_data.sh # checks model-ready inputs; does not rebuild raw data
|
||||
bash scripts/02_fit_model.sh
|
||||
bash scripts/03_extract_estimates.sh
|
||||
bash scripts/04_enrich_estimates.sh
|
||||
bash scripts/05_validate_estimates.sh
|
||||
```
|
||||
|
||||
The numbered scripts can also be run manually in that order. `scripts/05_validate_estimates.sh` runs validation checks after estimates have been generated.
|
||||
The numbered scripts can also be run manually in that order.
|
||||
|
||||
The Bayesian model is computationally expensive. The production run used 4 cores on an AMD Ryzen 9 7945HX and took 60,372 seconds, approximately 16 hours 46 minutes.
|
||||
|
||||
@@ -40,7 +41,7 @@ If model output is already available, rebuild estimates without refitting Stan:
|
||||
bash run_estimation.sh reuse
|
||||
```
|
||||
|
||||
`reuse` reruns source-data processing, post-estimation, and enrichment while skipping the Stan fitting step.
|
||||
`reuse` verifies the model-ready inputs, then reruns post-estimation, enrichment, and validation while skipping the Stan fitting step.
|
||||
|
||||
To check the local setup without fitting the model, run:
|
||||
|
||||
@@ -50,9 +51,15 @@ bash run_estimation.sh dry-run
|
||||
|
||||
## Data inputs
|
||||
|
||||
The model-ready inputs are included under `data/`.
|
||||
The model-ready inputs are included under `data/`:
|
||||
|
||||
Original raw source files are not redistributed. See `docs/RAW_DATA_SOURCES.md` for the list of original data sources, access information, and expected local filenames for regenerating the processed inputs.
|
||||
- `text_data.csv`
|
||||
- `expert.csv`
|
||||
- `lr_data.csv`
|
||||
- `union_mapping.csv`
|
||||
- `party_families.csv`
|
||||
|
||||
Original raw source files are not redistributed. See `docs/RAW_DATA_SOURCES.md` and `data-setup/` for the list of original data sources, access information, and expected local filenames for regenerating the processed inputs. Rebuilding inputs from raw files is separate from the normal estimation workflow and writes only to ignored `_local/` test directories; it never replaces committed `data/` inputs automatically.
|
||||
|
||||
## Output variables
|
||||
|
||||
|
||||
Reference in New Issue
Block a user