Make data setup a no-option entry point

This commit is contained in:
Armin
2026-06-15 17:53:31 +02:00
parent 9b925f340f
commit 55b4895872
4 changed files with 15 additions and 89 deletions
+2
View File
@@ -3,3 +3,5 @@
- For repositories on `git.seimel.app`, use the existing Tea/Gitea token from `~/.config/tea/config.yml` for authenticated `git push`/`git fetch` operations.
- Do not print, log, or expose the token.
- Prefer token-authenticated HTTPS over unauthenticated HTTPS or SSH when pushing to `git.seimel.app`.
- Python is forbidden for this project. Public workflow code and documentation must use R, Julia, Stan, and shell only; do not add Python scripts or Python references.
- `data-setup/run_data_setup.sh` is intentionally a no-option public entry point. It downloads script-accessible sources, checks local raw files, rebuilds generated inputs under `_local/`, compares them to committed `data/`, and never replaces committed inputs automatically.
+4 -37
View File
@@ -55,25 +55,10 @@ Run the full source-data setup workflow with:
bash data-setup/run_data_setup.sh
```
Maintenance options:
- `--dry-run` — check setup scripts only.
- `--download-only` — download script-accessible sources.
- `--build-test` — rebuild model-ready inputs from existing local sources.
- `--compare` — compare regenerated inputs with committed `data/` inputs.
- no option — download, rebuild, and compare locally.
Minimal check:
```bash
bash data-setup/run_data_setup.sh --dry-run
```
Download sources:
```bash
bash data-setup/run_data_setup.sh --download-only
```
The command downloads script-accessible sources, checks required local files,
rebuilds model-ready inputs under `_local/generated-inputs/`, and writes a
comparison report under `_local/reports/`. It never replaces committed files in
`data/`.
V-Party:
@@ -96,24 +81,6 @@ Morgan is not a public provider download. The local OCR/transcription file can b
_local/raw/morgan/morgan_positions_raw.csv
```
Rebuild model-ready inputs:
```bash
bash data-setup/run_data_setup.sh --build-test
```
Compare regenerated inputs:
```bash
bash data-setup/run_data_setup.sh --compare
```
Full local source test:
```bash
bash data-setup/run_data_setup.sh
```
The comparison writes `_local/reports/input_comparison.md`. Replacing committed inputs, if ever needed, is a separate manual decision and is not done by these scripts.
Known behavior: the current public/rebuilt sources run through the workflow successfully, but regenerated `text_data.csv`, `expert.csv`, and `lr_data.csv` are not byte-identical to the committed model-ready inputs because of source-version and linkage differences. The comparison report records those differences explicitly.
+1 -38
View File
@@ -4,32 +4,17 @@ set -euo pipefail
usage() {
cat >&2 <<'EOF'
Usage: bash data-setup/run_data_setup.sh
Optional maintenance modes:
--dry-run check setup scripts only
--download-only download source files where scriptable
--build-test rebuild inputs from existing local sources
--compare compare regenerated inputs with committed data/
EOF
}
if [ "$#" -gt 1 ]; then
if [ "$#" -ne 0 ]; then
usage
exit 1
fi
MODE="${1:---full-test}"
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd -P)"
cd "$repo_root"
case "$MODE" in
--dry-run|--download-only|--build-test|--compare|--full-test) ;;
*)
usage
exit 1
;;
esac
export PARTY2D_RAW_DATA_DIR="${PARTY2D_RAW_DATA_DIR:-$repo_root/_local/raw}"
export PARTY2D_BUILD_DIR="${PARTY2D_BUILD_DIR:-$repo_root/_local/build}"
export PARTY2D_GENERATED_INPUT_DIR="${PARTY2D_GENERATED_INPUT_DIR:-$repo_root/_local/generated-inputs}"
@@ -37,35 +22,13 @@ export PARTY2D_REPORT_DIR="${PARTY2D_REPORT_DIR:-$repo_root/_local/reports}"
export R_LIBS_USER="${R_LIBS_USER:-$repo_root/_local/R/library}"
mkdir -p "$R_LIBS_USER"
if [ "$MODE" = "--dry-run" ]; then
echo "Checking data-setup scripts only; no downloads and no rebuild."
command -v bash >/dev/null
command -v Rscript >/dev/null
bash -n data-setup/check_raw_data.sh
Rscript -e 'files <- list.files("data-setup/R", pattern="\\.R$", full.names=TRUE); invisible(lapply(files, parse)); cat("R setup scripts parse OK\n")'
echo "Data setup dry run passed."
exit 0
fi
if [ "$MODE" = "--compare" ]; then
Rscript data-setup/R/03_compare_generated_inputs.R
exit 0
fi
if [ "$MODE" = "--download-only" ]; then
Rscript data-setup/R/00_install_dependencies.R
Rscript data-setup/R/01_download_sources.R
exit 0
fi
if [ "$MODE" = "--full-test" ]; then
Rscript data-setup/R/00_install_dependencies.R
Rscript data-setup/R/01_download_sources.R || true
fi
bash data-setup/check_raw_data.sh
rm -rf "$PARTY2D_BUILD_DIR" "$PARTY2D_GENERATED_INPUT_DIR"
mkdir -p "$PARTY2D_BUILD_DIR" "$PARTY2D_GENERATED_INPUT_DIR"
Rscript data-setup/R/00_install_dependencies.R
Rscript data-setup/R/02_build_model_inputs.R
Rscript data-setup/R/03_compare_generated_inputs.R
+3 -9
View File
@@ -61,21 +61,15 @@ Observed checksum after download on 2026-06-11:
sha256 2cd8c9108b1b0b9c1b6594bb21acee709c70259cd02f450bc69fc09b505fc9fb
```
## Preflight check
## Local source check and rebuild
From the repository root, run:
```bash
bash data-setup/run_data_setup.sh --dry-run
```
To check local raw file placement and print byte sizes/checksums, run:
To check local raw file placement and print byte sizes/checksums directly, run:
```bash
bash data-setup/check_raw_data.sh
```
After restricted/manual files are in place, run the full local rebuild and comparison test with:
Run the full source-data setup, rebuild, and comparison test with:
```bash
bash data-setup/run_data_setup.sh