# Monitoring and debug output

## `RunMonitor`

The integrators report progress through a `RunMonitor`. You choose the
backend with the `monitor` argument or the `runmonitor_backend` setting:

| Backend | Output |
|---|---|
| `"print"` (default) | One line per window on stdout, and a summary |
| `"log"` | The same information as `logging` records on the `"tracmass"` logger |
| `"rich"` | A live terminal dashboard with a scrolling table of windows |

```python
tracmass.time_dependent(state, grid, lazy, monitor="rich")   # built and closed for you
tracmass.time_dependent(state, grid, lazy, monitor=None)     # no progress output

with tracmass.RunMonitor(backend="rich", max_table_rows=20) as mon:
    tracmass.time_dependent(state, grid, lazy, monitor=mon)  # your own instance
```

A window line of the `print` backend reads:

```text
012/180 2010-06-12 98213 active  stopped:41 (land:3, out_of_bounds_h:38), #sub:143, wall:7.1s [io:690ms jax:6.2s py:210ms]
```

| Part | Meaning |
|---|---|
| `012/180` | Window number / total windows |
| `2010-06-12` | Model date |
| `98213 active` | Particles still running after the window |
| `stopped:41 (...)` | Particles that stopped in this window, by reason |
| `#sub:143` | Sub-cycles used; `[LIMIT]` if `max_sub_iter_per_window` was reached |
| `wall`, `io`, `jax`, `py` | Wall-clock time in total, reading fluxes, in the kernel, and in Python |

The summary line splits the total time the same way and lists all stopped
particles by reason. The first window's `jax` time includes compiling the
kernel.

## Verbose debug output

When a run seems stuck, for example the monitor never appears, turn on
verbose output. Each stage is logged to stderr with the time and how long
it took: adapter calls, the size of each window's flux read and the time
per variable, seeding, the particle count, the resolved integrator options,
the monitor start, and each window's I/O, advection and `update_flux`
times.

```python
result = tracmass.run(env="gog", verbose=True)   # just this run
tracmass.set_verbose()                           # rest of the session
tracmass.set_verbose(False)                      # off again
```

```toml
[gog]
verbose = true    # also used when calling the integrators directly
```

```text
11:19:38 tracmass.driver: run: 4148000 particles seeded; calling time_dependent
11:19:38 tracmass.integrate.full_grid: time_dependent: loading first window (before monitor starts) ...
11:19:38 tracmass.gcm.flux_grid: compute: reading flux window dtm=2007-06-01 00:00:00, 42.48 GB, grid (4318, 3059, 50)
11:20:03 tracmass.gcm.flux_grid: compute:   uflux 2 x (4318, 3059, 50) float64: 25.1 s
```

This is the standard `logging` logger `"tracmass"` at `DEBUG` level, so the
usual logging configuration works too. Whether or not `verbose` is on,
reading more than 8 GB of flux data in one window raises a `UserWarning`
that suggests an `[env.gcm] subset`.