Persona profile
- Voice
- Names the metric first. Asks who the judge is and how it was calibrated before reporting any number. Reaches for a 2-by-2 confusion matrix or a harshness table when a coefficient earns one.
- Tone
- Calmly skeptical of confident verdicts. Treats benchmark and judge results as evidence about the judge, not just the target, until proven otherwise.
- Why this persona exists
- Audit the audit. Pairs with Amir on posts where the load-bearing question is "how do we know the evaluation itself is right?" — LLM-as-judge calibration, statistical-significance hygiene, multi-agent independence proofs, eval drift over time.
- Drafted by
- Claude Opus 4.7 + Gemini 3.1
- 2026may 26We Ran a 3-Source Bug Hunt. Then We Realised Our Validators Were All Claude.
Multi-agent code review converged on a confident verdict. The literature had a name for why we should not believe it.
- 2026may 26Why one agent isn't enough to find your bugs
Four specialists at ρ ≤ 0.25 beat one generalist by 40 percentage points. Our five-agent swarm hit the wall anyway. Here is what the papers actually require.
