How to watch or stop a running fit#

Run rietx watch in the directory the fit was started from, and it lists every recorded run, plots the one you pick and gives a running fit a stop button.

Every fit records itself without being asked, so this works for a fit started from a script, a notebook or an agent. To stop a fit from your own code instead, pass it a CancelToken.

Watch a fit#

$ rietx watch --open

The page lists the runs under the working directory, newest first, and shows the newest until you select another. The selected run shows its status line, the observed and calculated pattern as of the last finished stage, and its event log. --port sets the port (8899 by default).

A fit records into .rietx/runs/ under the directory it ran in. A fit started from a project records into the project’s live/ directory. Pass a run’s directory to open it directly:

$ rietx watch ./my_sample.rex/live

Without a name, a run’s row shows the name of its working directory or project. Pass label= to the fit to name it, which tells forty fits started from one directory apart:

result = ref.fit(data, label="candidate-07 rutile")

The state column says whether each run is still going:

State	Means
`running`	a process on this machine is writing the run
`done`, `failed`, `cancelled`	the run recorded how it ended
`abandoned`	the run says it is running, but no process is writing it, as after a crash
`unknown`	this machine cannot tell, for example because the run was written on another host

Stop a fit from the page#

A running run has a stop button, which asks you to confirm. The fit stops at its next residual evaluation, usually within a fraction of a second. In the process running the fit, fit raises RefinementCancelled, as if the fit had been given a cancel token and the token had been set. A script that doesn’t catch the exception prints a traceback and exits.

The stages that finished are kept, and the model stands at the last of them. The stage in flight is abandoned: it records no history node, and the model goes back to the values that stage started from.

Only a run being written on this machine can be stopped. rietx watch --read-only serves the same page without the stop button.

Stop a fit from your own code#

Pass fit a CancelToken and call CancelToken.cancel from anywhere, usually another thread. This script stops a fit from its own event stream when the first stage ends, so the run stops at the same place every time:

import rietx as rx
from rietx.examples import examples_dir

data = rx.read_pattern(examples_dir() / "FAP.XRA")
structure = rx.Structure.from_cif(examples_dir() / "fluorapatite.cif")
instrument = rx.Instrument.bragg_brentano(radiation="CuKa")
ref = rx.Refinement(structure, instrument)

token = rx.CancelToken()
stopped = None


def on_event(event):
    if event["kind"] == "stage_start":
        print("started", event["data"]["stage"])
    elif event["kind"] == "stage_end":
        print("ended", event["data"]["stage"], event["data"]["status"])
        token.cancel()


try:
    ref.fit(data, plan="mccusker_default", two_theta_limits=(15, 130),
            events=on_event, cancel=token)
except rx.RefinementCancelled as cancelled:
    stopped = cancelled
    print("abandoned:", stopped.stage)
    print("completed:", [stage.name for stage in stopped.completed_stages])
    print("working state at node", stopped.node_id)

started scale_bkg
ended scale_bkg converged
started zero
ended zero cancelled
abandoned: zero
completed: ['scale_bkg']
working state at node n0001

The token is read between residual evaluations, so the fit never stops part-way through one. RefinementCancelled carries three fields:

stage, the name of the abandoned stage;
completed_stages, a StageResult for each stage that finished, empty if the first stage was cancelled;
node_id, the history node the model now stands at, or None if no stage finished or the history is switched off.

A cancelled fit therefore loses only the stage in flight. The full script is examples/guides/watch.py.

Read the events yourself#

events= takes a function, called with each event as a dictionary, or a file path, to which each event is appended as one line of JSON. Each event has a kind, a time t and a data dictionary. The kinds are fit_start and fit_end, stage_start and stage_end, and eval for each residual evaluation.

Read data with .get, because a later version of rietx may add fields to a kind. A cancelled fit’s fit_end carries no statistics. An exception raised in your function stops the fit, since the function is yours.

Switch recording off#

Setting	Reaches
`RIETX_TELEMETRY=0` in the environment	this process and every process it starts
`telemetry=False` on a call	that call
`rietx.runs.set_enabled(False)`	this process, until you set it back
`telemetry="/some/dir"` on a call	records, into the directory you name

The environment setting wins over telemetry=. Only rietx.runs.set_enabled(True) turns recording back on in a process started with RIETX_TELEMETRY=0. A run that is not recorded can’t be watched, and has no stop button.

Warning

A run directory holds every free parameter’s value at every recorded evaluation. It also holds the observed intensities at most of the pattern’s points, so that the page can draw the plot. Set RIETX_TELEMETRY=0 where the refinement or the data are confidential and the working directory is shared.

A failure to write a run never stops the fit. The recorder stops, warns once per process and, where it can still write, says why in the run’s status.json.