What almost fooled my eval · Kasra Aminiyeganeh

My agents end every reply with a warrior send-off. “Godspeed Spartan.” It’s a silly line in my setup, and it became the most useful instrument in my last eval.

That eval tested the tip of making coding agents write in ASD-STE100, the English of aircraft manuals. This post is about the machinery: an eval for agents running inside their own CLIs, and the traps it nearly fell into. The whole harness is in the repo, ready to copy.

How each reply is made and scored: task, fresh folder with the rule file, headless CLI, reply with receipts, then two blind judges, a reading quiz and rule metrics, combined in a paired comparison

Each cell is one reply: a task modeled on agent work (the scenarios are invented), in an empty folder holding the rule under test, answered by a headless CLI in one turn. Three scorers look at it, and a paired comparison puts the scores together. Two rounds made 224 replies, and the Claude side of everything came to about $22 at API prices.

Trap	What gave it away	What I do now
My rules leak in	A send-off in a bare run	Count a canary
The agent acts	Codex searched the web	Tools off, actions logged
Judges have favorites	Results flip by judge	Two families must agree
The bill hides	Thinking tripled	Log usage per run
Noise poses as effect	Same cell, different score	Pair, bootstrap, rerun

My rules leak in

An agent CLI loads more than your prompt: user rules, project rules, an output style, hooks, memories. Round two needed none of mine. Claude Code’s --bare flag wants an API key, so I used --setting-sources project,local, and Codex got a CODEX_HOME holding only my Codex sign-in.

The send-off proves it worked. In round one, Codex signed off in all 64 replies, and Claude in at least 23 of 64. In the bare runs, zero of 96. A canary you can count beats a config you believe.

The agent acts

In the smoke test, Codex answered a Docker question by searching Docker’s docs, while Claude had no web access. So tools went off for both, and every prompt asks for an answer from what the model already knows. Each run also keeps the CLI’s raw event log. In all 224 replies, the only thing either model did was answer.

Judges have favorites

Two judges, Claude Opus and gpt-6.1-sol, scored every reply blind, ticking a hidden checklist of facts item by item. Quick guess before you open this: with the one line, did Claude’s replies get better or worse?

What the two judges said

One line minus bare (overall, out of 10)	Opus judge	sol judge
Claude’s replies	-0.71*	+0.17
Codex’s replies	-0.92*	-0.79*

It depends on who you ask. The asterisk means the 95% interval excludes zero. The Opus judge rates Claude’s replies about a point above Codex’s in general, and the sol judge leans the other way. With one judge, I’d have published whichever answer it gave.

So I only claim what both judges back: Codex gets worse, and Claude stays unresolved.

The bill hides

Visible replies barely changed length. Thinking tripled on Claude and rose 87% on Codex. Every run logs thinking tokens and seconds next to the text, which is the only reason I saw it. The token savings other people reported came from low-effort runs, where there’s barely any thinking to grow.

Noise poses as an effect

Two runs of the same cell differed by 0.56 to 0.76 points out of 10, about the size of the effects I was hunting. So every comparison is paired by task, run and judge, every difference gets a bootstrap 95% interval, and round two ran each cell three times.

The license wrote round two

You can’t paste the standard’s rules into a prompt, so round one used paraphrases. That raised the question round two answered with four words, Speak only in ASD-STE100: the models already knew it.

Steal it

The harness can test any line you’re thinking of adding to CLAUDE.md or AGENTS.md. Copy experiments/example, put your line in, and look before you spend:

export STE_EXP=experiments/my-rule
python3 harness/run.py --tasks T1 --reps 1 --dry   # list the cells, no model calls
python3 harness/run.py --tasks T1 --reps 1         # smoke test, then read one reply

If the smoke test shows a tool call, or a habit of yours you never gave it, something leaked. METHOD.md has the full design, down to the judge prompt and every flag.

This is a single-turn eval, without tools, on two models at one effort level, scored by model judges. Scoring a whole agent session is the harder job, and this doesn’t do it. If there’s a line in your rule file you’ve never measured, give it an afternoon and tell me in the issues what it did.