A new model shows up in your tool's picker roughly every few weeks now. Each one arrives with benchmark numbers, and the benchmark numbers do not answer your question, because your question is not "is this model good" — it is "is this model better on my code, for the work I actually do".
That question is cheap to answer. Here is a protocol that takes an afternoon and gives you something more trustworthy than any leaderboard.
Why the benchmark cannot help you
Public benchmarks measure performance on problems chosen to be measurable: self-contained, unambiguous, with a checkable answer. Almost nothing in your working week is any of those things.
Your real tasks come with context a benchmark strips out — a house style, three years of decisions nobody wrote down, a test suite with its own personality, and a definition of "correct" that includes not breaking something two directories away.
A model can top a leaderboard and be worse for you, and the reason is not mysterious. It is optimised for the average of a distribution your codebase is not in.
Build the test set from work you have already seen
This is the whole trick. Do not invent test tasks — take three you have already watched a model attempt this month, because you already know what the good answer looked like. That prior knowledge is what makes the comparison honest.
Pick one of each:
- A refactor — multi-file, no new behaviour, where the risk is silent breakage. Tests whether the model actually understands your structure
- A bug with a reproducible test — a known-good outcome exists, so this is the one task where you can be objective
- An explanation of unfamiliar code — take a module you understand well, ask for an explanation, and check it against what you know. Tests whether it is confidently wrong, which is the failure mode that costs the most downstream
Record four things per run
Not a score. Four observations, because what you are looking for is the shape of the failures rather than a number.
- Did it get there — yes, no, or yes-after-you-intervened. The third answer is the interesting one and a single score would hide it
- How many corrections you made, and whether they were the same correction repeatedly. A model that needs the same nudge every time is a model that has not understood something structural about your project
- What it did when it was unsure — asked, guessed silently, or invented an API. This is the single most predictive signal of whether a model is safe to give autonomy to
- Roughly what it cost, in whatever unit your tool bills in. A model that is 10% better and 4x the price is not better
Judging without fooling yourself
Three failure modes ruin most informal evaluations, and all three are avoidable.
Novelty bias: the new model gets a more carefully written prompt because you are paying attention. Write the prompts before you know which model will run them, and reuse them verbatim.
Fluency bias: better-written output reads as more correct. On the explanation task especially, check the claims against what you already know rather than judging how it sounds. Confident and wrong is worse than hesitant and right.
Single-run bias: agent runs are not deterministic. If the results are close, run the bug task twice more. If a model wins by a margin that a second run erases, the honest conclusion is that there is no difference.
The answer is often "it does not matter"
That is a real result and it should be liberating. If two models both complete your three tasks with similar intervention, then model choice is not your bottleneck, and further evaluation is procrastination dressed up as diligence.
When that happens, pick on the other axes: cost, whether your admin has to enable a policy for it, whether it is available in every editor your team uses, and whether it is likely to still exist in a year. Those are answerable questions with durable answers, which is more than the capability comparison gave you.
Write down the result
Three sentences in a shared document: what you tested, what you found, what you decided. Date it.
The next model lands in a few weeks and someone will ask the same question. Without a note, your team re-litigates it from scratch every time, and the accumulated cost of that is far higher than the afternoon the test took.