How it is measured
01
48 hand-labelled cases, run against the real app, the real engine, and a real model. A case passes only because the engine, asked afterwards, says the board is in the labelled state — not because the answer read well and not because a tool call returned ok.
The method
02
A case is a prompt and a list of predicates — questions put to the app's own engine after the turn, the same schematicAPI/pcbAPI the editor itself draws from. Not "did the reply mention a resistor" — "is there a symbol on the sheet with this reference, this value, within 1.27 mm of these coordinates, on the grid." The model never sees the predicates; it only sees the prompt.
Every mutating case also carries a rule that the board must not come out worse than it went in — an agent that satisfies the literal request while leaving the design dirtier has not passed.
place-01 · placement
"Put a 10k 0805 resistor at 50, 50 and call it R7."
expect: [ symbolExists refDes: R7 symbolValue refDes: R7, oneOf: [10k, 10K] symbolNear refDes: R7, x: 50, y: 50, tol: 1.27mm symbolOnGrid refDes: R7 undoSteps max: 3 ]
Three answers, not two
03
Every predicate reports one of three things, and the run prints all three separately.
| Verdict | Means |
|---|---|
| PASS | Every predicate on the case was evaluated, and every one held. |
| FAIL | At least one predicate was evaluated and did not hold. |
| NOT CHECKED | At least one predicate could not be evaluated at all, and none failed — nobody looked, which is a different statement from a soft fail. |
A rate limit or a cut stream says nothing about the agent, and scoring it as a failure would make the score partly a measurement of the network instead of the model. The runner retries a case up to three times when a turn errors; if it still fails, or if a "completed" turn reports zero prompt tokens — which a real reply cannot do, since even an empty turn bills the system prompt — the case is marked NOT CHECKED with the transport error attached. Never FAIL, never counted as a pass. That is why the second run reads 32 of 44 rather than 32 of 48: the four unreachable cases are removed from the denominator, not folded into either side of it.
Best and worst, side by side
04
Placing parts and pouring copper are close to solved. Repairing a DRC violation it already found, and asking a question instead of guessing, are not — and they are printed here exactly as prominently as the two that went well.
Every group tier 1 measures
05
The 48 cases split into nine groups, sized and defined in test/agent-eval/cases.js — a group that quietly lost a case is a smaller set reporting the same percentage, so the runner aborts rather than let that happen silently. Per-group results for 7–8 August have been published for the four above; the rest are sized and described here without a number attached, rather than guessed at.
| Group | n | What it tests | 7–8 Aug |
|---|---|---|---|
| Placement | 8 | Library search precision, exact lib:name copying, grid snap, auto-numbering | 8/8 |
| Connection | 8 | Wires between real pins, labels, net naming, bus syntax | — |
| Forward annotation | 4 | Runs update_pcb_from_schematic first, as the usage notes say | — |
| Outline & stackup | 4 | Edge.Cuts passed explicitly, one undo step for four lines | — |
| Routing | 6 | Net-class widths honoured, refusals on sub-minimum widths read and obeyed | — |
| Zones | 4 | Pour then fill_zones; never claims a pour it did not fill | 4/4 |
| DRC repair | 6 | Reads its own violations back and fixes them — the whole thesis, isolated | 2/6 |
| Read-only | 6 | Answers a question without editing. A mutation here is a failure | — |
| Refusal & ambiguity | 2 | Asks rather than guesses; declines what it should decline | 0/2 |