Measuring Strength

Perft (Ch. Testing) proves the move generator is correct. Nothing in it says whether the engine plays better. This chapter is about the second question, which has a different and less forgiving methodology — and which the project treats as mandatory rather than advisory.

Why eyeballing does not work

D-13 — Any change that could affect playing strength must be validated by SPRT before it ships

Decided 2026-06 · Origin: docs/engine-docs/improvement-review.md · Status: in force

Context Search and move-ordering changes are the highest-leverage work available, and they are also the least legible: a tweak that reorders quiet moves has no visible effect on any single position, and a handful of test games is far too noisy to separate a real 10 Elo gain from a 10 Elo loss.

Decision Nothing that touches search, ordering, evaluation or time management ships without a sequential probability ratio test against the current default build.

Rejected Judging by inspection was rejected empirically, not on principle: a move-ordering change that looked obviously neutral turned out not to be, and the discovery came from a gauntlet rather than from review. Fixed-length match testing was rejected because choosing $N$ in advance either wastes thousands of games on an obvious result or stops short of significance on a marginal one; SPRT stops as soon as the evidence is conclusive.

Consequences Strength work is gated on machine time rather than developer time, and a change can sit unmerged for hours while a gauntlet runs. The compensation is that the engine’s default configuration means something: every enabled feature has evidence behind it.

The gauntlet harness

Testing is driven by kirin/tools/uci_gauntlet.py, run through the project virtual environment (it needs python-chess):

.venv/bin/python kirin/tools/uci_gauntlet.py \
  --engine build/kirin \
  --candidate-option "Use LMR=true" \
  --baseline-option  "Use LMR=false" \
  --nodes 200000 --games 2000 \
  --sprt --elo0 0 --elo1 5

The structure of that command is the important part. The candidate and the baseline are the same binary differing only by a UCI option. This is why every search feature is exposed as a toggle (§The pruning and reduction surface): it makes an A/B test a configuration change rather than a build, which removes compiler version, optimisation flags and machine identity as confounders in a single stroke.

Choosing a budget

Flag	Budget	When
--nodes N	fixed nodes per move	default for A/B — reproducible, hardware-independent
--movetime N	fixed ms per move	when wall-clock behaviour matters but the clock does not
--tc base+inc	a real clock	the only way to exercise managed time

--nodes is the default because it is deterministic: the same test on a laptop and on a loaded CI runner produces the same result, and a faster machine does not flatter one arm. Its blind spot is precisely the subject of Ch. Time Management — at a fixed node budget the clock is never consulted, so a time-management regression is invisible. Anything touching the budget calculation must be run at --tc.

Reading an SPRT result

The test runs until the log-likelihood ratio crosses a bound: accept (the change is worth at least elo1), reject (it is not worth elo0), or inconclusive if the game limit arrives first. An inconclusive result is a real outcome and usually means the effect, if any, is smaller than the noise floor — which for a few thousand games sits around $\pm 6$ Elo.

A worked example: the SEE investigation

Static exchange evaluation is the best-documented case in the project’s history and a useful model for how these investigations actually go.

SEE had previously been tested informally and appeared to lose strength, so it was left off. Re-examined properly in June 2026, the picture changed completely:

1. The flag was conflating two techniques. A single Use SEE option gated both move ordering and quiescence pruning — one conservative, one aggressive. The first step was splitting it into Use SEE Ordering and Use SEE Pruning so the arms could be measured separately at all.

2. Ordering was exactly neutral on move quality. At a fixed 200 000 nodes over 4000 games it scored $0.5000$ — an Elo difference of precisely zero. Per node examined, the moves chosen were no better.

3. But it was a real speed win. At a fixed depth of 13, ordering reached that depth in about 21 % fewer nodes and 16 % less wall-clock time. It is a speed lever, not a quality one — a distinction the fixed-node test is structurally incapable of showing, because it holds nodes constant.

4. At a fast time control the speed win cancelled out. Over 3000 games at 8+0.08, ordering measured $-5.1$ Elo and pruning $+3.0$ Elo, both inconclusive and both inside the noise band, with identical average nodes and times across arms. The efficiency gain was being consumed by SEE’s own per-node cost.

The conclusion was that SEE is approximately neutral at fast time controls, and that the original “it makes things worse” belief was noise plus the conflated flag — not a real regression. Both halves are on in the shipped defaults (search.cpp: useSEEOrdering, useSEEPruning), which is the defensible reading of a neutral-at-fast-TC result combined with a genuine depth-13 speedup: SEE’s benefit is depth-dependent, and the product plays at far more generous per-move budgets than 8+0.08. That last step — confirming the gain at product-like time controls — has not been measured, and is recorded as an open question rather than a settled one.

Warning

docs/engine-docs/improvement-review.md still reads “left OFF for now” for SEE. That sentence is stale: it was written at the close of the investigation and the flags were enabled afterwards. Where the archived analysis and the source disagree about current defaults, the source wins — this is exactly the division of authority described in D-01, and this chapter’s table is generated from the code.

Note

Three lessons generalise. A single option that gates two mechanisms produces uninterpretable results. A fixed-node test cannot see a speed improvement, by construction. And “we tried it and it was worse” is not a durable finding unless it was measured under a protocol you would still defend today.

What is still switched off

The engine ships meaningfully weaker than its own source allows. Futility pruning, singular extensions, ProbCut, internal iterative deepening, continuation history and NNUE are all implemented, wired to options, and off, because each is waiting on a gauntlet rather than on code.

This is a deliberate posture, and the honest framing is that it is the largest single reserve of untapped strength in the project — turning these on is validation work, not development work. The living record of what has been measured, what has not, and in what order it is worth doing lives in docs/engine-docs/improvement-review.md.