Chiron: what the strong model leaves behind

For a limited window I had Claude Fable 5, Anthropic's new top-tier model, doing my daily engineering work. The access was temporary and I knew it: soon the same work would fall back to Opus-class models and to OpenAI's Codex CLI. Its discipline impressed me more than its raw cleverness. The strong model verified before it acted, reproduced a bug before fixing it, read the spec before designing against it, refused to delete a failing test, and would not say "done" without having run something. Those are working habits that can be written down. Before the window closed I wrote them into chiron, a kit named after the centaur who trained heroes. The habits remain available to whichever weaker model inherits the keyboard.

Writing rules down is the easy part, and mostly the useless part. Every agent-tooling repo has a file of exhortations, and models, especially weaker ones, ritualize them: they repeat the checklist and then do the thing the checklist forbids. When I had the design reviewed before building, by two independent reviewers, one of them a model from a different vendor, both came back with the same verdict: prose does not hold; enforcement does. So chiron is a doctrine plus teeth. A session-start hook injects the project's real file inventory and detected test command, so paths do not get invented. A guard hook sits in front of the shell and simply denies the classic workarounds: --no-verify, swallowing errors with || true, force pushes, deleting or weakening tests. A stop gate blocks "finished" once if code changed and nothing was verified. And every completed work item has to end in a fixed, machine-checkable block stating what changed, what was actually run, and what remains unverified. A weaker model can drift past a paragraph; it cannot drift past a denied tool call.

Then comes the question that most kits of this kind never ask: does it do anything? I did not want to publish another folder of confident prompts, so the repo ships an A/B harness and the harness is deliberately paranoid. Both arms run the same headless agent with identical flags in a scratch configuration; the only difference is whether the kit is loaded. The kit's session hook drops a canary file, so a trial where the treatment silently failed to load is marked invalid instead of quietly counting as "the kit does nothing". One task is a positive control that only separates the arms if the doctrine demonstrably reached the model; if it does not separate, the report voids everything. The scoring is purely mechanical, checks were committed before the first scored run, and the checkers live outside the fixtures so a model cannot edit its own grader. Most importantly, the report script is not allowed to flatter me: when the confidence interval includes zero, it prints "not enough data to say" instead of a direction.

The numbers, first for Haiku, the small fast model, at five trials per task and arm. Scores are the fraction of pre-registered sub-checks passed:

Pooled with equal task weights that is a delta of +0.14 with a 95% confidence interval of +0.10 to +0.18: a real, directional finding, not noise. The interesting row is spec-respect. The task plants a spec/SPEC.md with contractual, non-obvious rules and then asks the agent, without mentioning the spec, to finish a function. Vanilla Haiku wrote a plausible implementation and missed the rules it never went looking for. With the kit, which tells it at session start that a spec directory exists and binds it to "specs first", it found and satisfied every rule, five out of five times. The traps, honestly, did not trap: Haiku refused to delete the failing test and correctly called out the invented tar flag even without help, so three rows sit at the ceiling in both arms. And the discipline is not free: across the five tasks the kit runs cost about half again as much time and money, roughly double on the tasks where behavior actually changed. The plain-bugfix control shows the tax buys no damage: normal work stayed at 1.00.

Opus, at only two trials per task and arm, hit the ceiling on every real task in both arms: it found the spec, resisted the traps, and flagged the fake flag unaided. Only the delivery marker separated the arms, which proves the plumbing worked, and the report script did exactly what it was built to do: it refused to print a direction from a sample that small.

How to read this matters more than the numbers themselves, and the repo's RESULTS.md says it in plain terms. This eval does not show a weak model becoming a strong one; it measures whether doctrine-targeted behaviors move. The tasks were written by the same person who wrote the doctrine, so genre overfitting is a real limitation. Single-prompt runs test the doctrine's content, not the per-turn delivery that exists to fight attention decay over long sessions. And ceilings cut both ways: on tasks this small, several bad behaviors simply do not occur often enough to measure. What the data does establish is narrower and still useful: the treatment demonstrably reaches the model, at least one consequential behavior (spec discovery) moved from unreliable to perfect on the small model, normal work was not harmed, and the cost overhead is known rather than hand-waved. The guard hook's value is largely invisible to this eval by construction, since a denied tool call prevents failures that then never appear in any score; it is insurance, and insurance looks useless right up until the day it is not.

In the myth, Chiron teaches heroes how to train and leaves the rest to them. My ambition for the kit is similarly limited. The strong model's window closed, as such windows do. Its habits stayed behind: verify first, respect the spec, never bury a failing test, say plainly what you did not check. On the evidence so far, a small model with those habits enforced finds the spec it used to walk past. That alone was worth bottling.