aseimel
74 lines
1.7 KiB
Markdown
74 lines
1.7 KiB
Markdown
# Full model run operations
|
|
|
|
This note describes how to launch, monitor, and summarize the long full model
|
|
run that generates the posterior estimates.
|
|
|
|
Do not start a full run until raw-data and Stan-data preflight checks pass.
|
|
|
|
## Hardware note for runtime reporting
|
|
|
|
The production run should record elapsed time for this workstation as:
|
|
|
|
```text
|
|
Hardware: 4 cores of an AMD Ryzen 9 7945HX
|
|
```
|
|
|
|
After the run, record the wall-time reported by `src/sh/show_run_progress.sh` or
|
|
`outputs/model_outputs/latest/run_*/diagnostics/run_metrics.json`.
|
|
|
|
## Launch with durable logging
|
|
|
|
From the repository root:
|
|
|
|
```bash
|
|
mkdir -p outputs/logs
|
|
ts="$(date +%Y%m%d_%H%M%S)"
|
|
STAN_REFRESH=100 \
|
|
nohup bash run_estimation.sh full \
|
|
> "outputs/logs/full_run_${ts}.log" 2>&1 &
|
|
echo $! > "outputs/logs/full_run_${ts}.pid"
|
|
```
|
|
|
|
The wrapper configures local raw data, local R libraries, project-local temp
|
|
space, and the writable project-local CmdStan copy by default.
|
|
|
|
## Monitor during the run
|
|
|
|
Live log tail:
|
|
|
|
```bash
|
|
tail -f outputs/logs/full_run_<timestamp>.log
|
|
```
|
|
|
|
Convenience progress command:
|
|
|
|
```bash
|
|
bash src/sh/show_run_progress.sh
|
|
```
|
|
|
|
Stan progress is printed every `STAN_REFRESH` iterations (default `100`) and is
|
|
captured in the durable log.
|
|
|
|
## Inspect after completion
|
|
|
|
After a successful run, the chain CSVs are under:
|
|
|
|
```text
|
|
outputs/model_outputs/latest/run_<timestamp>/chains/
|
|
```
|
|
|
|
Run diagnostics and Stan logs are archived under:
|
|
|
|
```text
|
|
outputs/model_outputs/latest/run_<timestamp>/diagnostics/
|
|
```
|
|
|
|
Use:
|
|
|
|
```bash
|
|
bash src/sh/show_run_progress.sh
|
|
```
|
|
|
|
to print the latest run status, wall time, sampler configuration, divergences,
|
|
tree-depth hits, and command-configuration verification status.
|