Document raw source file reference
This commit is contained in:
@@ -0,0 +1,73 @@
|
||||
# Full model run operations
|
||||
|
||||
This note describes how to launch, monitor, and summarize the long full model
|
||||
run that generates the posterior estimates.
|
||||
|
||||
Do not start a full run until raw-data and Stan-data preflight checks pass.
|
||||
|
||||
## Hardware note for runtime reporting
|
||||
|
||||
The production run should record elapsed time for this workstation as:
|
||||
|
||||
```text
|
||||
Hardware: 4 cores of an AMD Ryzen 9 7945HX
|
||||
```
|
||||
|
||||
After the run, record the wall-time reported by `src/sh/show_run_progress.sh` or
|
||||
`outputs/model_outputs/latest/run_*/diagnostics/run_metrics.json`.
|
||||
|
||||
## Launch with durable logging
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
mkdir -p outputs/logs
|
||||
ts="$(date +%Y%m%d_%H%M%S)"
|
||||
STAN_REFRESH=100 \
|
||||
nohup bash run_estimation.sh full \
|
||||
> "outputs/logs/full_run_${ts}.log" 2>&1 &
|
||||
echo $! > "outputs/logs/full_run_${ts}.pid"
|
||||
```
|
||||
|
||||
The wrapper configures local raw data, local R libraries, project-local temp
|
||||
space, and the writable project-local CmdStan copy by default.
|
||||
|
||||
## Monitor during the run
|
||||
|
||||
Live log tail:
|
||||
|
||||
```bash
|
||||
tail -f outputs/logs/full_run_<timestamp>.log
|
||||
```
|
||||
|
||||
Convenience progress command:
|
||||
|
||||
```bash
|
||||
bash src/sh/show_run_progress.sh
|
||||
```
|
||||
|
||||
Stan progress is printed every `STAN_REFRESH` iterations (default `100`) and is
|
||||
captured in the durable log.
|
||||
|
||||
## Inspect after completion
|
||||
|
||||
After a successful run, the chain CSVs are under:
|
||||
|
||||
```text
|
||||
outputs/model_outputs/latest/run_<timestamp>/chains/
|
||||
```
|
||||
|
||||
Run diagnostics and Stan logs are archived under:
|
||||
|
||||
```text
|
||||
outputs/model_outputs/latest/run_<timestamp>/diagnostics/
|
||||
```
|
||||
|
||||
Use:
|
||||
|
||||
```bash
|
||||
bash src/sh/show_run_progress.sh
|
||||
```
|
||||
|
||||
to print the latest run status, wall time, sampler configuration, divergences,
|
||||
tree-depth hits, and command-configuration verification status.
|
||||
Reference in New Issue
Block a user