Make data setup a no-option entry point
This commit is contained in:
@@ -3,3 +3,5 @@
|
||||
- For repositories on `git.seimel.app`, use the existing Tea/Gitea token from `~/.config/tea/config.yml` for authenticated `git push`/`git fetch` operations.
|
||||
- Do not print, log, or expose the token.
|
||||
- Prefer token-authenticated HTTPS over unauthenticated HTTPS or SSH when pushing to `git.seimel.app`.
|
||||
- Python is forbidden for this project. Public workflow code and documentation must use R, Julia, Stan, and shell only; do not add Python scripts or Python references.
|
||||
- `data-setup/run_data_setup.sh` is intentionally a no-option public entry point. It downloads script-accessible sources, checks local raw files, rebuilds generated inputs under `_local/`, compares them to committed `data/`, and never replaces committed inputs automatically.
|
||||
|
||||
+4
-37
@@ -55,25 +55,10 @@ Run the full source-data setup workflow with:
|
||||
bash data-setup/run_data_setup.sh
|
||||
```
|
||||
|
||||
Maintenance options:
|
||||
|
||||
- `--dry-run` — check setup scripts only.
|
||||
- `--download-only` — download script-accessible sources.
|
||||
- `--build-test` — rebuild model-ready inputs from existing local sources.
|
||||
- `--compare` — compare regenerated inputs with committed `data/` inputs.
|
||||
- no option — download, rebuild, and compare locally.
|
||||
|
||||
Minimal check:
|
||||
|
||||
```bash
|
||||
bash data-setup/run_data_setup.sh --dry-run
|
||||
```
|
||||
|
||||
Download sources:
|
||||
|
||||
```bash
|
||||
bash data-setup/run_data_setup.sh --download-only
|
||||
```
|
||||
The command downloads script-accessible sources, checks required local files,
|
||||
rebuilds model-ready inputs under `_local/generated-inputs/`, and writes a
|
||||
comparison report under `_local/reports/`. It never replaces committed files in
|
||||
`data/`.
|
||||
|
||||
V-Party:
|
||||
|
||||
@@ -96,24 +81,6 @@ Morgan is not a public provider download. The local OCR/transcription file can b
|
||||
_local/raw/morgan/morgan_positions_raw.csv
|
||||
```
|
||||
|
||||
Rebuild model-ready inputs:
|
||||
|
||||
```bash
|
||||
bash data-setup/run_data_setup.sh --build-test
|
||||
```
|
||||
|
||||
Compare regenerated inputs:
|
||||
|
||||
```bash
|
||||
bash data-setup/run_data_setup.sh --compare
|
||||
```
|
||||
|
||||
Full local source test:
|
||||
|
||||
```bash
|
||||
bash data-setup/run_data_setup.sh
|
||||
```
|
||||
|
||||
The comparison writes `_local/reports/input_comparison.md`. Replacing committed inputs, if ever needed, is a separate manual decision and is not done by these scripts.
|
||||
|
||||
Known behavior: the current public/rebuilt sources run through the workflow successfully, but regenerated `text_data.csv`, `expert.csv`, and `lr_data.csv` are not byte-identical to the committed model-ready inputs because of source-version and linkage differences. The comparison report records those differences explicitly.
|
||||
|
||||
@@ -4,32 +4,17 @@ set -euo pipefail
|
||||
usage() {
|
||||
cat >&2 <<'EOF'
|
||||
Usage: bash data-setup/run_data_setup.sh
|
||||
|
||||
Optional maintenance modes:
|
||||
--dry-run check setup scripts only
|
||||
--download-only download source files where scriptable
|
||||
--build-test rebuild inputs from existing local sources
|
||||
--compare compare regenerated inputs with committed data/
|
||||
EOF
|
||||
}
|
||||
|
||||
if [ "$#" -gt 1 ]; then
|
||||
if [ "$#" -ne 0 ]; then
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
|
||||
MODE="${1:---full-test}"
|
||||
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd -P)"
|
||||
cd "$repo_root"
|
||||
|
||||
case "$MODE" in
|
||||
--dry-run|--download-only|--build-test|--compare|--full-test) ;;
|
||||
*)
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
export PARTY2D_RAW_DATA_DIR="${PARTY2D_RAW_DATA_DIR:-$repo_root/_local/raw}"
|
||||
export PARTY2D_BUILD_DIR="${PARTY2D_BUILD_DIR:-$repo_root/_local/build}"
|
||||
export PARTY2D_GENERATED_INPUT_DIR="${PARTY2D_GENERATED_INPUT_DIR:-$repo_root/_local/generated-inputs}"
|
||||
@@ -37,35 +22,13 @@ export PARTY2D_REPORT_DIR="${PARTY2D_REPORT_DIR:-$repo_root/_local/reports}"
|
||||
export R_LIBS_USER="${R_LIBS_USER:-$repo_root/_local/R/library}"
|
||||
mkdir -p "$R_LIBS_USER"
|
||||
|
||||
if [ "$MODE" = "--dry-run" ]; then
|
||||
echo "Checking data-setup scripts only; no downloads and no rebuild."
|
||||
command -v bash >/dev/null
|
||||
command -v Rscript >/dev/null
|
||||
bash -n data-setup/check_raw_data.sh
|
||||
Rscript -e 'files <- list.files("data-setup/R", pattern="\\.R$", full.names=TRUE); invisible(lapply(files, parse)); cat("R setup scripts parse OK\n")'
|
||||
echo "Data setup dry run passed."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ "$MODE" = "--compare" ]; then
|
||||
Rscript data-setup/R/03_compare_generated_inputs.R
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ "$MODE" = "--download-only" ]; then
|
||||
Rscript data-setup/R/00_install_dependencies.R
|
||||
Rscript data-setup/R/01_download_sources.R
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ "$MODE" = "--full-test" ]; then
|
||||
Rscript data-setup/R/00_install_dependencies.R
|
||||
Rscript data-setup/R/01_download_sources.R || true
|
||||
fi
|
||||
|
||||
command -v bash >/dev/null
|
||||
command -v Rscript >/dev/null
|
||||
bash -n data-setup/check_raw_data.sh
|
||||
Rscript data-setup/R/00_install_dependencies.R
|
||||
Rscript data-setup/R/01_download_sources.R || true
|
||||
bash data-setup/check_raw_data.sh
|
||||
rm -rf "$PARTY2D_BUILD_DIR" "$PARTY2D_GENERATED_INPUT_DIR"
|
||||
mkdir -p "$PARTY2D_BUILD_DIR" "$PARTY2D_GENERATED_INPUT_DIR"
|
||||
Rscript data-setup/R/00_install_dependencies.R
|
||||
Rscript data-setup/R/02_build_model_inputs.R
|
||||
Rscript data-setup/R/03_compare_generated_inputs.R
|
||||
|
||||
@@ -61,21 +61,15 @@ Observed checksum after download on 2026-06-11:
|
||||
sha256 2cd8c9108b1b0b9c1b6594bb21acee709c70259cd02f450bc69fc09b505fc9fb
|
||||
```
|
||||
|
||||
## Preflight check
|
||||
## Local source check and rebuild
|
||||
|
||||
From the repository root, run:
|
||||
|
||||
```bash
|
||||
bash data-setup/run_data_setup.sh --dry-run
|
||||
```
|
||||
|
||||
To check local raw file placement and print byte sizes/checksums, run:
|
||||
To check local raw file placement and print byte sizes/checksums directly, run:
|
||||
|
||||
```bash
|
||||
bash data-setup/check_raw_data.sh
|
||||
```
|
||||
|
||||
After restricted/manual files are in place, run the full local rebuild and comparison test with:
|
||||
Run the full source-data setup, rebuild, and comparison test with:
|
||||
|
||||
```bash
|
||||
bash data-setup/run_data_setup.sh
|
||||
|
||||
Reference in New Issue
Block a user