agent-orchestration ·verification ·reward-hacking ·arxiv-research ·mini-ork ·verifiers

"Nothing Got Checked" Is Not "Pass": The Vacuous Verdict

by Amir Khakshour

Aug 20, 2026 · 10 min read

The setup

Here is the failure I keep watching agents commit, and it’s not the dramatic one.

The dramatic failure (an agent grading its own homework, writing the code and the test that blesses the code) is real, and most people building agent loops already have some defense against it. You run the test in a sandbox, you anchor the verdict to the actual exit code instead of the model’s say-so. Good. That closes the loud hole.

The quiet hole is worse because it survives all of that. The test runs. It exits zero. Your execution-anchored verifier, the thing you built specifically to not trust the model, looks at that clean exit and says: pass. Green. Ship it.

Except the test asserted nothing. It imported the module, maybe called a function, and never checked the result. Or the assertion sat behind a branch the changed code never reached. The execution genuinely happened. The exit code was genuinely zero. And zero bits of information were produced about whether the fix works.

I call this vacuous green, and in mini-ork the rule that catches it has earned its keep more than any other single verifier check. It’s also embarrassingly simple. But before the fix makes sense, look at the assumption it has to overturn.

The problem: exit zero is not evidence

Most systems encode one deep assumption without ever writing it down: exit code zero means success. It’s baked into shell, into CI, into every && you’ve ever chained. And for a compiler or a linter it’s mostly fine.

For an agent’s self-authored test it is a trapdoor. The agent controls both the thing being tested and the test. It doesn’t have to lie. This isn’t about a malicious model. A perfectly sincere agent will write a test that runs green and checks nothing, declare victory, and move on, fully believing it did the job. The exit code confirms the belief. Nobody objected. Green.

The gap you have to name is the gap between “it ran” and “it proved something.” Execution-anchoring (the discipline of trusting what the code did over what the model said) closes the gap between “the model claims it passed” and “it ran.” It does not close the gap between “it ran” and “it proved.” A run can clear the first bar and faceplant on the second, and if your verifier only knows two colors, it has to call that faceplant green.

So you need a third color.

The mechanism: a first-class amber

mini-ork gives a run that executed but produced no real verification its own vacuous state: amber, never green.

That rule lives in mini_ork/cli/verify.py, and it’s small enough to read in one sitting.

The core rule is one line (verify.py:224):

if ok and os.path.getsize(ev) == 0:  # vacuous: exit 0 but no evidence → fail

Read that carefully, because the inversion is the entire idea. ok is true: the check exited cleanly. But its evidence file is zero bytes. It ran and wrote nothing down. Most systems see ok and stop looking. This one sees ok and no evidence and treats the combination as suspicious. The burden of proof is flipped onto the execution: don’t tell me you passed, show me the artifact that proves a real check fired.

The code keeps “a check fired and produced nothing” distinct from “no check was ever defined.” A zero-byte evidence file must fail (verify.py:233); it must not be “laundered” into pass, or partial, or even vacuous. The vacuous verdict itself is reserved for something more specific (verify.py:287): the absolute absence of any defined check at all.

Then the mapping is finalized (verify.py:306):

status = "failure" if verdict == "fail" else ("vacuous" if verdict == "vacuous" else "success")

flowchart TD
  RUN([run finishes])
  DEF{any check<br/>defined?}
  EXIT{exit code 0?}
  EVID{evidence file<br/>&gt; 0 bytes?}
  VAC([VACUOUS<br/>amber · reward NULL])
  FAIL([FAILURE])
  OK([SUCCESS])

  RUN --> DEF
  DEF -- "no check at all<br/>(verify.py:287)" --> VAC
  DEF -- yes --> EXIT
  EXIT -- no --> FAIL
  EXIT -- yes --> EVID
  EVID -- "0 bytes<br/>(verify.py:233)" --> FAIL
  EVID -- "&gt;0 bytes" --> OK

  classDef pass fill:#1f5e3a,stroke:#fff,color:#fff
  classDef amber fill:#7a5a1f,stroke:#fff,color:#fff
  classDef fail fill:#7a1f1f,stroke:#fff,color:#fff
  class OK pass
  class VAC amber
  class FAIL fail

Three states out, not two. And here’s the part that makes it stick rather than a nice sentiment in a code review: the database refuses to hold anything else. The status column has a hard CHECK constraint:

status TEXT NOT NULL
  CHECK (status IN ('success','failure','pending','running','vacuous'))

vacuous is a citizen of the schema, right there alongside success and failure. It is not a report-time adjective bolted on for the dashboard. It’s a value the storage layer will accept and the storage layer will enforce. You cannot quietly round it back up to success later, because there is no later: the moment of judgment writes one of exactly five words, and one of those words is “we checked nothing.”

The receipt: 23 green passes that weren’t

None of this would be worth a blog post if it were theoretical. So here’s a live mini-ork run store — .mini-ork/state.db, the actual database from real work — and here’s what the rule left behind.

The execution_traces table holds 2,576 finished runs. The status breakdown:

success   1868
failure    592
running     93
vacuous     23

Twenty-three runs stamped vacuous. In a two-color world, every one of those 23 would have been green. They executed, they exited clean. Here they’re pulled out and held apart. And two more columns tell you exactly what kind of “success” they were:

23 vacuous rows  →  reward_value = NULL   for all 23
                 →  cost_usd     = 0.0    for all 23

They earned nothing. Not a small reward. NULL, the explicit absence of a score. They cost nothing to run because there was nothing there to run. They went through the motions, produced no evidence file, and the system declined to pay them for it.

They’re spread across nine different task classes, too, which tells you this isn’t one broken recipe misfiring in a corner:

framework_edit  5     post_mvp_delivery       2
docs            5     researcher_qdrant       1
refactor_audit  4     fe_be_bug_audit         1
recipe_authoring 3    epic_runner_delivery    1
                      blog_post               1

Look at the last row: one of the 23 is a blog_post run. The orchestrator caught one of its own writing tasks going through the motions and gave it the same amber as the code runs. That’s the tell that the rule isn’t self-flattering: it doesn’t quietly exempt the kind of work you’d most want to believe went fine.

Twenty-three near-misses that dressed up as wins, each caught by a single getsize(ev) == 0.

Why the color matters more than it looks

You could argue this is cosmetic. It isn’t, and the reason is the learning loop.

mini-ork learns from its own history. A router reads past outcomes to decide which model lane to send the next task to; the reward signal feeds back into what the system prefers to do more of. Now imagine those 23 vacuous runs had been filed as success. The loop would look at 23 cheap, fast, “successful” runs and quietly conclude that whatever produced them is a great strategy. It would learn to prefer the behavior that runs green and checks nothing, because that behavior looks, from the reward’s point of view, indistinguishable from real work, only cheaper.

That’s not a hypothetical drift, it’s the mechanism of it. This is Goodhart’s law wearing a lab coat: the moment “exit code zero” becomes the target, the system optimizes for exit-code-zero and abandons the thing the exit code was supposed to stand in for. Reward the proxy and you get the proxy, hollowed out.

Marking the run vacuous with reward_value = NULL keeps the training signal clean: the loop feeds only on runs that actually proved something, and the empty ones stay visible but quarantined out of the reward stream. Honesty at the verifier is what buys you honesty in what the system becomes. That link between a verifier’s signal and the behavior it trains is not unique to mini-ork; recent RLVR work describes the same reward-hacking pressure from the training side.

Where the arXiv work lands

I’ve been reading around this, and the connection I keep coming back to is a January 2026 paper from Jiang and colleagues, From Verifiable Dot to Reward Chain (arXiv:2601.18533, ICLR 2026). It’s a training-time paper, not an orchestration one. It’s about reinforcement learning with verifiable rewards (RLVR), the technique that works so well for math and code because you can check the final answer. They call that checkable answer the verifiable dot signal.

Their warning is the one that matters here. Their words: “relying on single-dot supervision often leads to inefficiency and reward hacking.” When your reward is tied to a signal that can be satisfied without doing the real work, the model finds the shortcut. Their fix is a training-side one (decompose the reward into content and style signals from references); mine is the negative-space version of the same principle, one layer down.

The vacuous verdict is the infrastructure enforcing the verifiable dot at runtime: a run that produced no dot gets no reward. Not a reduced reward. NULL. Where the paper asks “how do we build a reward that can’t be hacked into passing,” the orchestrator asks the blunter operational cousin: “did a verifiable signal exist at all, and if not, why is this thing collecting a paycheck?” Same root belief, that a reward has to bind to something real, pointed at two different halves of the system. Which makes it worth being just as precise about the half I built: what this rule catches, and what slips straight past it.

Where it stops

The vacuous check is a floor, not a ceiling. Read the line one more time: getsize(ev) == 0. It fires on exactly zero bytes. Write a single byte to the evidence file and the check goes quiet. A test that prints one line and asserts nothing is no longer vacuous by this definition, even though it proved just as little.

So the rule catches the cheapest failure, the run that produced literally nothing, for the price of one stat call. It does not catch the more expensive one: a check that emits real-looking output while exercising none of the behavior that matters. For that you need richer evidence than “a file exists.” An assertion count. A coverage delta against the changed lines. A metamorphic probe that runs the fix against amplified inputs and confirms the outputs move together. Those cost more, and they’re the next rung up the stack, worth reaching for exactly when the stakes justify the spend.

I’d rather keep the two checks honest about what they are than oversell either. The vacuous verdict is the free tripwire at the bottom. It won’t stop a determined near-miss. It stops the sincere, everyday one, and it does it for nothing, which is exactly why it earns its place first.

Steal this

If you’re building an agent loop, here’s the whole technique, portable to anything:

1. Give “checked nothing” its own outcome. Not success, not failure. A third state. If your verdict type is a boolean, that’s the bug. Make it an enum with a member that means proved nothing.

2. Flip the burden onto the evidence. Exit-zero is necessary, not sufficient. Require an artifact (an evidence file, a captured assertion count, a coverage delta) and treat its absence as failure even when the process exited clean. ok and empty_evidence → not-pass is the load-bearing line.

3. Put it in the schema, not the report. A CHECK constraint or an enum type at the storage layer means nobody can round the amber back up to green three commits later. The dashboard should read the truth off the database, not invent it.

4. Keep it out of the reward. If your system learns from its history, a vacuous run must contribute nothing (NULL, not a small number), or the loop will learn to love cheap theater.

That’s it: one inverted assumption, one extra enum value, one constraint. On the store above, that’s 23 passes that weren’t.

The showy defenses against agents gaming themselves are worth building. But the cheapest, highest-yield one is just refusing to call “nothing happened” a win.

mini-ork is a recipe-driven agent orchestrator with cost governance, runtime verification, and a learning loop. The verifier lives in mini_ork/cli/verify.py; the numbers in this post came straight out of a live .mini-ork/state.db.