The question
A lot of small, self-hosted AI agents work the same way: the model does a task, then a second step checks the work and asks for a fix if something looks wrong. It's a cheap way to catch mistakes without paying for a bigger model.
We wanted to know whether that actually works, on the kind of local, open-weight models people run themselves rather than the largest hosted ones. And if it does work, whether it's a better use of your compute budget than the obvious alternative: skip the checking step and just run a stronger model in the first place.
What we found
Checking the work does help. Across 300 pre-registered test problems, a model with an independent verifier watching over it got more answers right than the same model working alone (51.7% versus 45.3%). That gain held up statistically (p = 0.016).
But it doesn't help nearly as much as switching to a better model. A stronger model working completely alone, with no verifier at all, solved 75.0% of the same problems. That's not a close race. The gap between "small model plus a verifier" and "bigger model, no help needed" was decisive (p = 1.7 x 10^-12), and it held whenever we tried this comparison, including when we controlled for the extra compute the checking step costs.
We also tested a more generous version of the setup, where the verifier is allowed to just hand over the correct answer instead of a hint. We expected this to recover meaningfully more of the errors than an ordinary hint does. It didn't, not by a margin we could call real (p = 0.073). Being told exactly what's wrong doesn't reliably help a model fix its own mistake if the model wasn't capable of getting there on its own.
Why this happens
The pattern underneath all three results is the same: the bottleneck is the model's ability to generate a correct answer, not the checker's ability to spot a wrong one.
We confirmed this a second way, using a good verifier paired with a weaker generator at the same overall scale as the strong-model comparison above. The verifier did its job well: it correctly flagged 15 of 17 wrong answers. But the generator, told exactly what was wrong, still couldn't fix them. Zero of those 15 flagged errors got corrected. A verifier can tell you a problem is wrong. It cannot make a model capable of solving it.
If you're deciding where to spend a compute budget on a small agentic loop, this says: spend it on a better generator before you spend it on a better checker. The checker is worth having, since it's a real, if modest, improvement over nothing. It just isn't a substitute for capability the model doesn't have.
One thing we deliberately left out: we also tried this on code-generation tasks and the weaker model failed almost everything outright, with or without help. That's a real finding about that model on that task, not a finding about verification, so it isn't part of the headline result here. We're treating it as a separate question for later.
A note on process
This is our own result, from our own experiments, on models we ran ourselves. The methodology was pre-registered before we ran the 300-problem test, which is why we can report a clean p-value rather than one found by looking at the data afterward.
What it hasn't had yet is a qualified second pair of eyes. Marek is starting a postgraduate AI programme this September, and we'd rather have him revisit the full technical paper with that training behind him than publish it as-is now. This summary is what we're comfortable standing behind today. The full paper, methods, and data are available on request at hello@carronlabs.co.uk in the meantime.
Back to all research