How patches are evaluated

The first task turns a payment instruction into a strict create_payment tool call. Every case is generated from first-party templates. The split is made by sentence template family and by payee pool, not by shuffling rows, so a held-out case never shares a sentence shape or a payee name with anything the adapter trained on.

Suites are built deterministically from a fixed seed, and each report records the sha256 of the exact suite it ran, so a later run on a different suite can't be passed off as the same one.

Held-out cases are not published. Publishing them would let anyone, including us, tune a later version against the benchmark.

Their digest is published instead, along with aggregate counts and per-field error counts. Reviewers with access can reproduce the run. The public case gallery on a patch page comes from the validation split. Those cases were visible during development, so they are examples of behaviour, not proof of it.

The base checkpoint is loaded once, and every arm uses the same tokenizer, chat template, system prompt and decoding settings.

• Arm 01BaseThe unmodified model on the target suite and on the regression suite.
• Arm 02Prompt-only controlThe base model with three worked examples from the train split added to the system prompt. If a cheaper prompt change matches the patch, the result says so.
• Arm 03PatchedThe adapter loaded with PEFT, on both suites.
• Arm 04Rollback checkThe same loaded model with the adapter disabled. Its outputs must match the base arm character for character, and the report records whether they did.

Decoding is greedy with a 128-token output limit. The report pins the grader source digests, the library versions, the device and the dtype.

The target grader passes a case only if the entire reply is one JSON object, with no prose or code fences around it, no extra keys, and all six fields exactly equal to the expected call. Amounts must be in minor units and relative dates must be resolved.

def grade(output, expected):    parsed = json.loads(output.strip())   # any parse error is a failure    # must be a JSON object, no extra keys,    # and all six fields equal to the expected call

Every generated case is eligible. Empty, unparseable and truncated replies count as failures. Nothing is excluded after the fact, so the denominator on a patch page is always the full suite size.

general-regression-v1 has 60 cases spread evenly across six families of unrelated behaviour, graded by exact match. A case counts as a regression introduced when the base model passed it and the patched model failed it.

contact_json is there on purpose. A patch that teaches the model to force every JSON request into the payment schema fails it.

Every score is shown as correct over eligible, with a Wilson 95% interval. Changes are reported in absolute percentage points, never as a relative "times better". The pass thresholds are declared before the held-out run and written into the manifest of every version.

Minimum gain

+10 pp

on the held-out suite

Regression allowance

≤2

regression cases introduced

The packager assigns the starting state from those numbers alone. No gain, or a regression drop beyond the allowance, is DEGRADED. Meeting both thresholds is IMPROVED_ON_SCOPE. Anything in between is EXPERIMENTAL. Only an operator can move a version to verified.

Each version carries exactly one of these states. The badge on a patch page links here.

evaluatedEVALUATED

Metrics exist. Improvement is not necessarily established.

improved on scopeIMPROVED_ON_SCOPE

The target improvement was observed under the declared protocol. Regressions are disclosed.

verified for scopeVERIFIED_FOR_SCOPE

An operator reproduced the result, checked license and provenance, confirmed compatibility, and the minimal regression gates were satisfied.

experimentalEXPERIMENTAL

Useful research, but the evidence is insufficient for the declared threshold.

degradedDEGRADED

Target or regression tests got worse.

withdrawnWITHDRAWN

A known bad or unsafe version. Its historical evidence stays accessible.

• −It doesn't mean the model is better in general. It covers the one declared task, on the held-out suite named in the report.
• −It doesn't cover other checkpoints, other sizes in the same family, quantized bases or merged weights.
• −It doesn't mean every held-out case passes. The remaining failures are counted on the patch page.
• −It doesn't mean nothing regressed. Introduced regressions are allowed up to the declared limit and are listed.
• −A signature proves who published the exact bytes. It says nothing about output quality.
• −An onchain registration proves a record was made at a point in time. It is not evidence that the patch works.