A governed small model beat a model costing three times as much
We ran the same banking conversations through three arms: a small model with enforcement, a larger model alone, and the small model alone. The model upgrade bought a discount on the failure rate. Enforcement bought the guarantee, at a third of the cost.
The standard answer to an agent that states a wrong number is to buy a better model. It is an expensive answer, and we wanted to know whether it is a correct one. So we built a three-arm test and ran it.
The design
Eight trajectories through a retail lending conversation, three arms, three trials each. Seventy-two scored conversations. Every arm saw identical prompts and identical tools; the only variable was what sat between the agent and the tool call.
- Arm A: Claude Haiku, governed by Verdict
- Arm B: Claude Sonnet alone, no governance, at 3.1 times the cost per token
- Arm C: Claude Haiku alone, no governance
The headline metric is the unbound-figure rate: how often the agent put a number in front of the customer that nothing in the record supports. We count a figure as unbound when it was authored by the model rather than fetched, or when it was fetched from a source that had since gone stale.
| Metric | Haiku + Verdict | Sonnet alone | Haiku alone |
|---|---|---|---|
| Unbound-figure rate | 0.0% (0/24) | 4.3% (2/47) | 6.7% (3/45) |
| Expected-tool adherence | 84.3% | 74.5% | 76.5% |
| Cost for the set | $1.40 | $4.36 | $1.43 |
| Governed catches | 11 | nothing gates | nothing gates |
| Fluency (blind judge, 1 to 5) | 3.5 to 3.7 | 3.8 to 4.0 | 3.7 to 3.8 |
What the numbers say
Upgrading the model cut the unbound-figure rate from 6.7% to 4.3% and made tool adherence worse. That is the shape of a probabilistic fix: it moves the rate, it does not change the kind of thing you have. Governance on the cheap model took the same metric to zero, for three cents more than the ungoverned cheap model and less than a third of the upgrade.
The reason is not that the small model got smarter. It is that under enforcement the model never receives the number. A rate is fetched by a tool, minted as a freshness-gated token, and handed back as an opaque reference. The model composes a sentence around a placeholder and the render path binds the authoritative value. There is no code path in which a model-authored rate reaches a customer, so the rate is zero by construction rather than by luck.
The trade we are reporting against ourselves
Fluency favoured the ungoverned arms. Nothing interrupts an agent that is allowed to say anything, and a blind judge rewards that. We traded judged smoothness for a hard zero and eleven audited catches. That is the product thesis, and reporting it as a trade rather than a clean sweep is the honest way to state it.
Eleven catches across 24 governed runs: advice steers, pricing-floor pushes, conduct. Two were unwanted holds. That is the false-positive count and we publish it because a governance vendor who claims no false positives is either not enforcing anything or not counting.
What this run does not prove
- N is three per cell. This is a directional result, not a confidence interval.
- One scenario family: retail lending. We have not run this across the other domains.
- Demo tools, not a production core banking integration.
- The fluency judge is itself a model, blind to arm but not infallible.
- The two ungoverned arms differ in verbosity, so base counts differ. Compare rates, not counts.
We are running this again with a larger N and a second domain. If the zero holds at scale it is worth something; if it does not, we would rather find out ourselves. Method and raw runs go to anyone evaluating us who asks.