diff --git a/AGENTS.md b/AGENTS.md index 711f699..8a014cf 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -3,3 +3,5 @@ - For repositories on `git.seimel.app`, use the existing Tea/Gitea token from `~/.config/tea/config.yml` for authenticated `git push`/`git fetch` operations. - Do not print, log, or expose the token. - Prefer token-authenticated HTTPS over unauthenticated HTTPS or SSH when pushing to `git.seimel.app`. +- Python is forbidden for this project. Public workflow code and documentation must use R, Julia, Stan, and shell only; do not add Python scripts or Python references. +- `data-setup/run_data_setup.sh` is intentionally a no-option public entry point. It downloads script-accessible sources, checks local raw files, rebuilds generated inputs under `_local/`, compares them to committed `data/`, and never replaces committed inputs automatically. diff --git a/data-setup/README.md b/data-setup/README.md index eb447ce..70b316c 100644 --- a/data-setup/README.md +++ b/data-setup/README.md @@ -55,25 +55,10 @@ Run the full source-data setup workflow with: bash data-setup/run_data_setup.sh ``` -Maintenance options: - -- `--dry-run` — check setup scripts only. -- `--download-only` — download script-accessible sources. -- `--build-test` — rebuild model-ready inputs from existing local sources. -- `--compare` — compare regenerated inputs with committed `data/` inputs. -- no option — download, rebuild, and compare locally. - -Minimal check: - -```bash -bash data-setup/run_data_setup.sh --dry-run -``` - -Download sources: - -```bash -bash data-setup/run_data_setup.sh --download-only -``` +The command downloads script-accessible sources, checks required local files, +rebuilds model-ready inputs under `_local/generated-inputs/`, and writes a +comparison report under `_local/reports/`. It never replaces committed files in +`data/`. V-Party: @@ -96,24 +81,6 @@ Morgan is not a public provider download. The local OCR/transcription file can b _local/raw/morgan/morgan_positions_raw.csv ``` -Rebuild model-ready inputs: - -```bash -bash data-setup/run_data_setup.sh --build-test -``` - -Compare regenerated inputs: - -```bash -bash data-setup/run_data_setup.sh --compare -``` - -Full local source test: - -```bash -bash data-setup/run_data_setup.sh -``` - The comparison writes `_local/reports/input_comparison.md`. Replacing committed inputs, if ever needed, is a separate manual decision and is not done by these scripts. Known behavior: the current public/rebuilt sources run through the workflow successfully, but regenerated `text_data.csv`, `expert.csv`, and `lr_data.csv` are not byte-identical to the committed model-ready inputs because of source-version and linkage differences. The comparison report records those differences explicitly. diff --git a/data-setup/run_data_setup.sh b/data-setup/run_data_setup.sh index 2d38a28..ca63099 100644 --- a/data-setup/run_data_setup.sh +++ b/data-setup/run_data_setup.sh @@ -4,32 +4,17 @@ set -euo pipefail usage() { cat >&2 <<'EOF' Usage: bash data-setup/run_data_setup.sh - -Optional maintenance modes: - --dry-run check setup scripts only - --download-only download source files where scriptable - --build-test rebuild inputs from existing local sources - --compare compare regenerated inputs with committed data/ EOF } -if [ "$#" -gt 1 ]; then +if [ "$#" -ne 0 ]; then usage exit 1 fi -MODE="${1:---full-test}" repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd -P)" cd "$repo_root" -case "$MODE" in - --dry-run|--download-only|--build-test|--compare|--full-test) ;; - *) - usage - exit 1 - ;; -esac - export PARTY2D_RAW_DATA_DIR="${PARTY2D_RAW_DATA_DIR:-$repo_root/_local/raw}" export PARTY2D_BUILD_DIR="${PARTY2D_BUILD_DIR:-$repo_root/_local/build}" export PARTY2D_GENERATED_INPUT_DIR="${PARTY2D_GENERATED_INPUT_DIR:-$repo_root/_local/generated-inputs}" @@ -37,35 +22,13 @@ export PARTY2D_REPORT_DIR="${PARTY2D_REPORT_DIR:-$repo_root/_local/reports}" export R_LIBS_USER="${R_LIBS_USER:-$repo_root/_local/R/library}" mkdir -p "$R_LIBS_USER" -if [ "$MODE" = "--dry-run" ]; then - echo "Checking data-setup scripts only; no downloads and no rebuild." - command -v bash >/dev/null - command -v Rscript >/dev/null - bash -n data-setup/check_raw_data.sh - Rscript -e 'files <- list.files("data-setup/R", pattern="\\.R$", full.names=TRUE); invisible(lapply(files, parse)); cat("R setup scripts parse OK\n")' - echo "Data setup dry run passed." - exit 0 -fi - -if [ "$MODE" = "--compare" ]; then - Rscript data-setup/R/03_compare_generated_inputs.R - exit 0 -fi - -if [ "$MODE" = "--download-only" ]; then - Rscript data-setup/R/00_install_dependencies.R - Rscript data-setup/R/01_download_sources.R - exit 0 -fi - -if [ "$MODE" = "--full-test" ]; then - Rscript data-setup/R/00_install_dependencies.R - Rscript data-setup/R/01_download_sources.R || true -fi - +command -v bash >/dev/null +command -v Rscript >/dev/null +bash -n data-setup/check_raw_data.sh +Rscript data-setup/R/00_install_dependencies.R +Rscript data-setup/R/01_download_sources.R || true bash data-setup/check_raw_data.sh rm -rf "$PARTY2D_BUILD_DIR" "$PARTY2D_GENERATED_INPUT_DIR" mkdir -p "$PARTY2D_BUILD_DIR" "$PARTY2D_GENERATED_INPUT_DIR" -Rscript data-setup/R/00_install_dependencies.R Rscript data-setup/R/02_build_model_inputs.R Rscript data-setup/R/03_compare_generated_inputs.R diff --git a/docs/RAW_DATA_SOURCES.md b/docs/RAW_DATA_SOURCES.md index 57c5cd4..61876d4 100644 --- a/docs/RAW_DATA_SOURCES.md +++ b/docs/RAW_DATA_SOURCES.md @@ -61,21 +61,15 @@ Observed checksum after download on 2026-06-11: sha256 2cd8c9108b1b0b9c1b6594bb21acee709c70259cd02f450bc69fc09b505fc9fb ``` -## Preflight check +## Local source check and rebuild -From the repository root, run: - -```bash -bash data-setup/run_data_setup.sh --dry-run -``` - -To check local raw file placement and print byte sizes/checksums, run: +To check local raw file placement and print byte sizes/checksums directly, run: ```bash bash data-setup/check_raw_data.sh ``` -After restricted/manual files are in place, run the full local rebuild and comparison test with: +Run the full source-data setup, rebuild, and comparison test with: ```bash bash data-setup/run_data_setup.sh