Monitoring and debug output#

RunMonitor#

The integrators report progress through a RunMonitor. You choose the backend with the monitor argument or the runmonitor_backend setting:

Backend

Output

"print" (default)

One line per window on stdout, and a summary

"log"

The same information as logging records on the "tracmass" logger

"rich"

A live terminal dashboard with a scrolling table of windows

 
tracmass.time_dependent(state, grid, lazy, monitor="rich")   # built and closed for you
tracmass.time_dependent(state, grid, lazy, monitor=None)     # no progress output

with tracmass.RunMonitor(backend="rich", max_table_rows=20) as mon:
    tracmass.time_dependent(state, grid, lazy, monitor=mon)  # your own instance

A window line of the print backend reads:

012/180 2010-06-12 98213 active  stopped:41 (land:3, out_of_bounds_h:38), #sub:143, wall:7.1s [io:690ms jax:6.2s py:210ms]

Part

Meaning

012/180

Window number / total windows

2010-06-12

Model date

98213 active

Particles still running after the window

stopped:41 (...)

Particles that stopped in this window, by reason

#sub:143

Sub-cycles used; [LIMIT] if max_sub_iter_per_window was reached

wall, io, jax, py

Wall-clock time in total, reading fluxes, in the kernel, and in Python

The summary line splits the total time the same way and lists all stopped particles by reason. The first window’s jax time includes compiling the kernel.

Verbose debug output#

When a run seems stuck, for example the monitor never appears, turn on verbose output. Each stage is logged to stderr with the time and how long it took: adapter calls, the size of each window’s flux read and the time per variable, seeding, the particle count, the resolved integrator options, the monitor start, and each window’s I/O, advection and update_flux times.

result = tracmass.run(env="gog", verbose=True)   # just this run
tracmass.set_verbose()                           # rest of the session
tracmass.set_verbose(False)                      # off again

[gog]
verbose = true    # also used when calling the integrators directly

11:19:38 tracmass.driver: run: 4148000 particles seeded; calling time_dependent
11:19:38 tracmass.integrate.full_grid: time_dependent: loading first window (before monitor starts) ...
11:19:38 tracmass.gcm.flux_grid: compute: reading flux window dtm=2007-06-01 00:00:00, 42.48 GB, grid (4318, 3059, 50)
11:20:03 tracmass.gcm.flux_grid: compute:   uflux 2 x (4318, 3059, 50) float64: 25.1 s

This is the standard logging logger "tracmass" at DEBUG level, so the usual logging configuration works too. Whether or not verbose is on, reading more than 8 GB of flux data in one window raises a UserWarning that suggests an [env.gcm] subset.