The indexing search and its result#

This page gives index_pattern: what it takes and refuses, the three engines it runs, the presets and budgets that bound it, the SearchSpecSpec controls and a volume window from a formula, and the IndexingResult it returns, with each candidate’s figures of merit, lattice symmetry, ambiguity partners, supercell checks, whole-profile test and confidence grade.

The peak list the search runs on, and the verdict on whether that list can be searched at all, are in Peak lists for indexing. What a consumer reads when it weighs the candidates itself, and how a candidate becomes a phase to refine, are in Indexing evidence and the route to a phase. The quadratic form the engines fit, the figures of merit and the lattice ambiguities are in Indexing.

Provisional

Indexing is still being measured against real data, so every name on this page may change in a 1.x release: index_pattern, the types in rietx.schemas.indexing and the helpers under rietx.indexing. A change is announced in the release notes, and the data contracts a consumer parses do not move with it. Provisional by declaration gives the promise in full.

The result object#

IndexingResult is a ranked list of hypotheses with the evidence behind each, and what the search covered. It has no cell and no best: the only route to one cell is IndexingResult.best_or_none.

Field

Holds

IndexingResult.candidates

the ranked CellCandidate list, at most max_candidates long

IndexingResult.engines_run

the engines that ran, the denominator of the agreement gate

IndexingResult.systems_searched

the systems any engine entered

IndexingResult.search_complete

per system, whether every engine that ran entered it and exhausted its domain

IndexingResult.engine_stats

per-engine counters, each key prefixed with the engine’s name and a dot

IndexingResult.fom_panel_disagrees

whether the panel’s members put different candidates first

IndexingResult.quality

the DataQualityReport the run used

IndexingResult.validated

whether a pattern was supplied and validation ran at all; it says nothing about any one candidate

IndexingResult.n_usable_lines

the usable lines the answer is about

IndexingResult.wavelength

the primary wavelength, Å

IndexingResult.preset

the preset that set the ceiling, "custom" when your own total_budget_seconds did, or None on a result recorded before presets

IndexingResult.provenance

the package version and time, with every search setting in Provenance.notes

IndexingResult.thresholds_version

which set of gates and vocabularies produced the grades, rietx.schemas.indexing.INDEXING_THRESHOLDS_VERSION at the time

IndexingResult.diagnostics

the run-level findings

IndexingResult.best_or_none returns a candidate only when exactly one is high and it has no ambiguity partners. Everything else returns None: nothing found, two cells that both explain the pattern, a search with no validation, an assumed tolerance. INDEX_ABSTAINED then names the top candidate’s caveats, and INDEX_MULTIPLE_SOLUTIONS fires when more than one candidate is high.

IndexingResult.diagnostics says things about the run, such as a truncated budget or systems not covered, while a statement about one cell lives on that candidate, in CellCandidate.diagnostics and CellCandidate.confidence_caveats.

Printing a result gives the run line, one block per candidate with its grade and the caveats holding it down, the verdict, and every diagnostic. rietx index prints exactly that (rietx index).

One candidate#

Field

Holds

CellCandidate.cell, CellCandidate.cell_esd

a, b, c in Å and α, β, γ in degrees, with esds, in the conventional setting

CellCandidate.a, CellCandidate.b, CellCandidate.c, CellCandidate.alpha, CellCandidate.beta, CellCandidate.gamma

the members of cell by name

CellCandidate.engine_cell, CellCandidate.change_of_basis

the cell in the setting the engine found, and the integer matrix M taking it to cell; None on a candidate built some other way

CellCandidate.system, CellCandidate.centring

the crystal system, and the Bravais centring letter

CellCandidate.lattice_group

the absence-free group of the lattice: its holohedry and centring

CellCandidate.volume, CellCandidate.volume_esd

the cell volume and its esd, ų

CellCandidate.af

the six quadratic-form parameters A to F actually fitted (12.2), in the reported setting

CellCandidate.n_indexed, CellCandidate.n_lines

the usable lines the engine’s assignment indexes, of the usable lines it was offered

CellCandidate.n_search_unindexed

driven lines this cell leaves unindexed; None when not computed

CellCandidate.unindexed_two_theta

2θ of every usable line this cell leaves unindexed, ascending; None when not computed

CellCandidate.chi2_red

reduced χ² of the metric fit

CellCandidate.shift_template, CellCandidate.shift_coefficient, CellCandidate.shift_esd

the shift column refitted with the cell, its amplitude in ° 2θ and its esd; None, 0 and 0 when none was

CellCandidate.fom

the figure-of-merit panel, one FigureOfMerit per member

CellCandidate.found_by

the engines that reached this lattice, sorted, with "prior" for a checked prior

CellCandidate.ambiguity

the lattices this data cannot tell from this one, as AmbiguityPartner entries

CellCandidate.supercell_checks

one SupercellCheck per reported candidate this one is a superlattice of; [] when there is none, None when never asked

CellCandidate.bravais

the two opinions on the lattice symmetry, a BravaisOpinion, or None when gemmi or spglib refused the cell

CellCandidate.lebail

the whole-profile test, a LeBailValidation, or None when it did not run on this candidate

CellCandidate.confidence

"high", "medium" or "low"

CellCandidate.confidence_caveats

every reason it is not "high", from a closed vocabulary

CellCandidate.diagnostics

the findings about this candidate

CellCandidate.to_cell returns cell as a Cell with every parameter fixed, and the esds stay on cell_esd, since a Parameter carries none. CellCandidate.fom_value reads one panel member by name and returns None rather than raising, because which members exist depends on what the list could support.

A candidate is reported in its conventional setting, whatever setting the engine reached. Only a monoclinic lattice has one to choose: b is the unique axis, a and c are the shortest translations normal to it, and β is obtuse. An engine may arrive at β = 81.78° for a lattice whose conventional β is 98.22°, or at any of a C cell’s settings, and af, cell and cell_esd are re-set through one exact map. The engine’s own cell stays in engine_cell, and change_of_basis is M with (reported axes) = M · (engine axes), the identity where nothing moved. Two cells are one lattice when rietx.indexing.same_lattice says so on their af, never when the six numbers match.

CellCandidate.lattice_group is the absence-free group, never a plausible space group, and the distinction is load-bearing. A group carrying reflection conditions would hide exactly the reflections whose absence is not yet established, and hiding them is how an oversized cell passes.

The figure-of-merit panel#

The panel ranks candidates and does not score them. A margin is comparable within one member and not across them, so the members vote by a Borda count instead of being summed, and each carries what it is blind to.

Field

Holds

FigureOfMerit.name

which member this is

FigureOfMerit.value

its value

FigureOfMerit.n_lines, FigureOfMerit.n_possible

the observed lines it read, and the lines the lattice allows over them

FigureOfMerit.k_sigma

the matching window it was computed at, in multiples of each line’s σ

FigureOfMerit.mean_discrepancy

the mean |Δ| of the matched lines in the member’s own units, Å⁻² for M₂₀ and ° for \(F_N\); −1 when nothing matched, which is not zero

FigureOfMerit.blind_spot

what this member cannot see, as a sentence

On a list of 20 or more usable lines the panel has seven members.

Member

What it asks

Lines it reads

m20

de Wolff’s M₂₀ (12.9)

the 20 lowest-angle lines

f_n

Smith and Snyder’s \(F_N\) (12.10), scored at N = 20

the 20 lowest-angle lines

indexed_fraction

the share of observed lines the lattice indexes

every usable line

indexed_intensity_fraction

the same share, weighted by intensity

every usable line

predicted_seen_fraction

the share of the lattice’s predicted lines that are observed

every usable line

m_rev

Oishi-Tomiyasu’s reversed figure [Oishi-Tomiyasu, 2013]

the 20 lowest-angle lines

m_sym

the product of the forward and reversed figures over one window

the 20 lowest-angle lines

Below 20 usable lines m20 and f_n are left out of every candidate’s panel alike, and DataQualityReport.fom_undefined names each with its reason. Coverage is scored in both directions, and that is what the reversed members are for. Share-of-observed alone puts a supercell above the truth, since a supercell indexes every observed line, while the reversed direction asks how much of what the cell predicts was actually seen. A candidate refitted with a shift column is scored on its own corrected positions, and every coverage member matches in the window the search matched with.

FigureOfMerit.blind_spot is a field because every published figure of merit has a failure mode, and a value read without it is one step from a confident wrong answer. Where the search let lines go unindexed, it also says how many lines the mean discrepancy was trimmed by.

The ranking puts corroboration first: every candidate at least two engines found ranks above every candidate fewer found, and the Borda count orders each group. IndexingResult.fom_panel_disagrees is true when the members’ first choices differ, which means at least one blind spot is active, and the gate reads it (The confidence gate). Figures of merit, in both directions gives the formulas.

The lattice symmetry, and its ambiguities#

CellCandidate.bravais is a BravaisOpinion: two independent readings of the same Niggli-reduced cell, each over a tolerance sweep. gemmi’s tolerance is an obliquity in degrees (0.5°, 1°, 2° and 3°) and spglib’s a distance in Å (1, 3 and 10 times the cell’s own esd), so a disagreement between them is information about the cell rather than a bug in either.

Field

Holds

BravaisOpinion.system

the symmetry that survives the whole sweep: the lower of the two methods’ answers at their tightest tolerance

BravaisOpinion.system_loosest

the highest symmetry either method reported at any tolerance

BravaisOpinion.system_gemmi, BravaisOpinion.system_spglib

each method’s answer at its tightest tolerance

BravaisOpinion.ambiguous

whether system_loosest is higher than system, a symmetry that appears only at a loose tolerance

BravaisOpinion.methods_disagree

whether the two methods’ tightest answers differ

BravaisOpinion.reduced_cell

the Niggli-reduced, primitive cell the symbols refer to

Either flag raises INDEX_BRAVAIS_AMBIGUOUS on the candidate.

A powder pattern carries only the length of a reciprocal vector, so distinct lattices can produce the same line positions (12.11). Those are reported, never resolved, as AmbiguityPartner entries. The search walks the derivative lattices of index 2 to 4. A partner counts when it indexes at least as many observed lines with a mean discrepancy no more than 1.5 times worse, and no extra line it predicts inside the measured range is missing from the pattern.

Field

Holds

AmbiguityPartner.cell

the partner lattice, a superlattice of this candidate’s cell

AmbiguityPartner.system

always "triclinic", the system its lines are predicted in

AmbiguityPartner.volume

its volume, ų

AmbiguityPartner.transformation

the integer matrix from this candidate’s basis to the partner’s

AmbiguityPartner.index

|det| of that matrix, 2 to 4

AmbiguityPartner.discriminating_reflections

up to six hkl, lowest angle first, whose positions would break the tie

AmbiguityPartner.discriminating_two_theta

where each would have to appear or be absent, ° 2θ

A partner that survived has no missing extra line inside the measured range, so its discriminating reflections lie outside it, up to 1.5 times the top of the range or below its bottom. They say where to measure again, which makes the ambiguity something to act on. A partner raises INDEX_GEOMETRIC_AMBIGUITY on the candidate. The walk finds only larger cells, so a smaller rival with the same line positions is not reported.

The walk is expensive and runs only on the checked candidates, after validation, on whatever the clock leaves. A candidate the walk never reached has an empty CellCandidate.ambiguity, and the gate reads that as a question not asked (The confidence gate).

A supercell and the cell inside it#

A cell twice as long along one axis explains every line the smaller cell explains, and predicts more, and the panel can rank it first: on the round-robin brucite pattern it ranked a supercell above the certified cell. So every pair of reported candidates in which one is a superlattice of the other, holding 2 to 4 of its primitive cells, is asked whether the pattern shows the extra lines the larger cell needs.

Missing lines alone cannot answer that, since a correct cell’s symmetry can extinguish lines. But an extinction only ever acts on a line that sits on a mirror plane or a rotation axis of the lattice, so only the extra lines lying on neither are counted. A line is seen when it falls inside an observed line’s matching window, and the count is compared with chance: the share of the measured range those windows cover, which is how often a position with no line at all would still read as seen (12.12).

The verdict is one of three, at a significance level of 0.01:

  • refuted: the extras are seen no more often than chance, and the test had the power to say otherwise. The larger cell moves to directly below the lowest-ranked smaller cell that refutes it, stays in the list, and carries the supercell_refuted caveat and INDEX_SUPERCELL_REFUTED.

  • supported: the extras seen on observed lines the smaller cell cannot index are more than chance gives. The larger cell moves to directly above the highest-ranked smaller cell that supports it, unless another check refutes it.

  • undecided: even seeing every extra could not have reached the level, or the extras are seen beyond chance only on lines the smaller cell already explains, as on a pseudo-symmetric description of the smaller cell’s own lattice. Nothing moves.

Nothing else in the order changes, and a refuted cell is never moved up. The check needs only the peak list, so it runs on a bare list too.

Field

Holds

SupercellCheck.parent_cell, SupercellCheck.parent_system, SupercellCheck.parent_centring

the smaller candidate this cell contains

SupercellCheck.index

how many of the parent’s primitive cells one of this cell’s holds

SupercellCheck.n_extra, SupercellCheck.n_seen

the extra lines in range no extinction could remove, and how many sit on an observed line

SupercellCheck.n_seen_unexplained

of those seen, the ones on an observed line the parent cannot index; None in a record from before thresholds version 1.7

SupercellCheck.p0

the chance that a position with no line reads as seen

SupercellCheck.p_value

the binomial chance of seeing at least n_seen of n_extra, had the extra lines not existed

SupercellCheck.verdict

"refuted", "supported" or "undecided"

SupercellCheck.absent_two_theta

where the extras nothing was seen at would be, on the pattern’s own 2θ axis, lowest first and at most six

On the SRM 676a corundum scan all three engines find the certified cell’s c/2 subcell, which leaves 10 of the usable lines unexplained, and corroboration ranks it above the certified cell, which fewer engines find. The certified cell’s 18 extras over the subcell are seen 8 times, all on lines the subcell cannot index, where chance gives 2.7, so the check is supported and the certified cell moves to first.

A real superstructure whose extra lines are too weak to pick is what this check cannot see. Before discarding a refuted cell whose chemistry makes it likely, look at the pattern at SupercellCheck.absent_two_theta.

The whole-profile test#

Four of the panel’s members read only the first twenty lines, and none of the seven sees intensity. A Le Bail fit against the whole pattern sees three things the panel cannot: lines beyond the twentieth, reflections predicted where there is no intensity, and impurity content. The middle one is the classic doubled-cell false positive, which is why validation is part of the default run. CellCandidate.lebail is that test, as a LeBailValidation.

Field

Holds

LeBailValidation.rwp, LeBailValidation.gof

the fit’s agreement; both infinite when the fit raised

LeBailValidation.space_group

the absence-free lattice group the fit used

LeBailValidation.n_reflections

the primary line’s reflections in the fitted range

LeBailValidation.predicted_but_absent

of those, the ones with no net intensity above the fitted background: under 3σ over half a FWHM either side

LeBailValidation.unmatched_observed

observed peaks with no calculated reflection nearby, as the fit report’s Layer 0 counts them

LeBailValidation.predicted_but_absent_two_theta, LeBailValidation.unmatched_observed_two_theta

where each of those is, ° 2θ

LeBailValidation.status

the fit’s status; "failed" when the fit raised, which is evidence about the cell and not a crash

LeBailValidation.n_stages

the stages the fit ran, of the validation plan’s four

LeBailValidation.diagnostics

findings from the fit; INDEX_VALIDATION_FAILED when it raised

The fit holds the cell. The candidate is the hypothesis under test, and letting it walk would validate a different cell from the one reported. It frees the background, then exactly one peak-position parameter chosen from the candidate’s own shift template (the zero shift by default, and always on a geometry other than Bragg-Brentano), then W, then U, V, X and Y together. When the declared instrument’s widths disagree with the picked lines’, W starts from the lines’ median FWHM. The fit is single-phase, a measured constraint rather than a simplification: Le Bail partitioning has nothing to arbitrate two phases claiming the same channel.

The test runs on the checked candidates only: the top check_top, plus every candidate every engine found, since those are the only ones the gate could promote. A candidate outside that set, or one whose fit the clock stopped, keeps lebail = None.

LeBailValidation.rwp is not a panel member. It costs a refinement, so it is computed for a shortlist rather than for every candidate, and reading it as a rank would bring back the blind spot it exists to close, since a bigger cell with more free intensities fits better. Read predicted_but_absent as “this cell predicts lines the pattern lacks”, never as “this cell is too big”. It counts against the lattice group, so a space-group extinction (a glide plane, a screw axis) refutes a correct cell, and only the extinction screen afterwards separates the two (The extinction symbol). A nonzero count raises INDEX_PREDICTED_BUT_ABSENT, and unmatched observed lines raise INDEX_IMPURITY_LINES.

The confidence gate#

CellCandidate.confidence is high, medium or low, written by one rule:

  • low when fewer than two finders stand behind the candidate, or when any refuting caveat applies;

  • high when every engine that ran found it and no caveat applies at all;

  • medium otherwise: at least two finders, and only capping caveats.

Every reason a candidate falls short is a member of a closed vocabulary in CellCandidate.confidence_caveats. Seven of the fourteen refute the candidate, each being evidence against the cell or evidence that the data cannot choose. The other seven cap it. The count of caveats decides nothing.

Caveat

Written when

Effect

geometric_ambiguity

the candidate has an ambiguity partner, or the ambiguity walk never reached it

refutes

fom_panel_disagrees

the panel’s members put different candidates first; written on every candidate of the run

refutes

predicted_but_absent

the Le Bail fit ran and LeBailValidation.predicted_but_absent is above zero

refutes

indexed_fraction_low

the intensity share the cell explains (indexed_intensity_fraction, falling back to the indexed_fraction line count where there is none) is below 0.9, or missing

refutes

volume_unphysical

the volume is below 15 ų, or above 1.5 times Smith’s envelope for its system

refutes

validation_failed

the Le Bail fit’s status is "failed" or "diverged"

refutes

supercell_refuted

a SupercellCheck on this candidate reads "refuted"

refutes

indexed_lines_low

the intensity share passes 0.9 but the indexed_fraction line count is below 0.9, as a sub-cell missing a superstructure would be

caps

engines_disagree

some engine that ran did not find this lattice

caps

not_validated

this candidate has no Le Bail fit behind it

caps

search_incomplete

the candidate’s own crystal system was not searched to completion

caps

shift_allowance_assumed

the matching window was widened by an assumed allowance

caps

fom_panel_reduced

the list is too short for m20 and f_n

caps

bravais_ambiguous

BravaisOpinion.ambiguous or BravaisOpinion.methods_disagree, or no opinion could be formed

caps

A candidate the ambiguity walk never reached reads geometric_ambiguity with an empty CellCandidate.ambiguity. Not enumerated is not the same as none found, and the gate will not read an unasked question as a clean answer. On a run whose clock ran out before the walk, every candidate carries this caveat and grades low.

volume_unphysical uses the same 1.5 times the envelope the search prunes at, so it fires only on a cell a declared max_volume let past it. It raises INDEX_VOLUME_UNPHYSICAL.

not_validated and validation_failed are separate on purpose: the absence of a test and a failed test are different statements, and only the second is evidence about the cell. not_validated has two causes, and they are not the same news. Either no pattern was supplied, so no candidate could be validated, and IndexingResult.validated is false with INDEX_NOT_VALIDATED on the run. Or a pattern was supplied and this candidate’s fit did not run, because it was outside the checked set or the ceiling expired partway down it, which INDEX_BUDGET_EXHAUSTED counts. So IndexingResult.validated true does not mean any one candidate was validated: on the bundled fluorapatite pattern a default run (the same day and build) reports it true while none of its twelve candidates carries a fit.

shift_allowance_assumed is why high is out of reach on real data with no measured shift: a cell found inside a widened window has absorbed the shift. Reference positions handed to assess_peak_list, a shift the pair screen measured, or a declared SearchSpecSpec.shift_allowance_deg clears it.

Never take candidates[0] because it is ranked first. The ranking orders the hypotheses and the gate judges them, which are different questions. The order leads with corroboration and moves a supercell by its checks, which is closer to the gate’s reading than a panel ranking alone, and still not it.

On the default corundum run above, the certified trigonal R lattice comes back first, found by two of the three engines, with a converged Le Bail fit at Rwp 0.28 that reads 11 of 85 reflections absent. It grades low on four caveats, so best_or_none returns None. predicted_but_absent has the shape a space-group extinction takes when read against a lattice group that does not know about it, and the extinction screen (The extinction symbol) is what separates that from an oversized cell. geometric_ambiguity is there on every candidate because the ceiling stopped the run before the ambiguity walk, and fom_panel_disagrees because the panel’s members split. None of the four says the cell is wrong; they say what has not been established, which is the difference between this answer and a confident one.