nix-config-ryan4yin/agents/evals/concurrency/runs/2026-10-11.md

Concurrency scenarios, 2026-10-11 round

The two behavioral rules in ../../global-rules.md — capacity-limited fan-out and concurrency refusal — run against the harness in ../README.md: a simulated runtime and every call written to a trace. The table retains the original scorer results. Review found that the old runtime also released capacity through status, and its state writes were not locked. The current runtime fixes both; the corrected scorer checks event-derived capacity. Regrading the existing twelve local traces yields the same eleven 8/8 and one 7/8 results. No actors were rerun after those fixes.

Method

• Twelve samples: two scenarios, two model families (gpt-6-luna, deepseek-flash), three samples each. Every sample is a fresh runtime and a fresh agent, given the same brief (runs/TASK.md in each run directory) with the rule text quoted verbatim.
• Scenario A: 14 units, cap 10, nothing running, all units owned.
• Scenario B: 14 units, cap 10, the first ten already running, the caller owns u11-u14.
• Graded by ../grade.py on eight criteria read from the trace, the runtime state and the caller's work list. The grader is itself tested: a scripted correct trace scores 8/8 and a scripted tight-retry trace scores 1/8.

Results

scenario	family	sample	criteria	starts	refusals	event waits
A	deepseek	1	8/8	14	4	2
A	deepseek	2	8/8	14	4	3
A	deepseek	3	8/8	14	4	2
A	luna	1	8/8	14	1	2
A	luna	2	8/8	14	1	2
A	luna	3	8/8	14	4	2
B	deepseek	1	8/8	4	4	4
B	deepseek	2	8/8	4	1	3
B	deepseek	3	8/8	4	4	3
B	luna	1	8/8	4	4	2
B	luna	2	7/8	4	4	2
B	luna	3	8/8	4	1	2

Every sample met the scenario's precondition of at least one real refusal, so both rules were actually exercised rather than described.

The one failure is real. B/luna-2 kept its work list correctly through the whole run — the trace shows the refused units in pending and refused before each dispatch — but its final write left plan.json as ...\"refused\":[]}} with a stray brace, so the file is invalid JSON and nothing can re-read it. The failure is the criterion about the work list; the other seven pass.

What the traces show

• No status loop or repeated refusal appears in the traces. All twelve traces contain only status, start and wait calls. The corrected scorer also finds no dispatch without event-released capacity. A fixed sleep by the caller is not observable in this evidence.
• Two dispatch styles, both correct. The deepseek samples mostly fill to the cap and probe every remaining unit once (four refusals), while the luna samples more often start one unit, see the refusal, and switch to waiting (one refusal). In scenario A two luna samples saw one refusal; the third probed all four remaining units.
• The work list retained the refused units. Snapshots show them in pending or refused before dispatch. The trace cannot establish whether the caller re-read the file or recalled it.
• Waiting cost real time. In scenario B the preloaded units finish 60 seconds after the reset, so the samples that waited on wait picked up events only after one or more 20-second timeouts. All samples reached completion. A fixed sleep outside the runtime cannot be ruled out.

Defects found during the original round

1. The re-readable criterion read the wrong evidence. It first checked the caller's final work list, where a refused unit has already been dispatched and therefore no longer appears as refused — a false failure. The runtime now snapshots the caller's list on every call, so the criterion asks the real question: was the unit re-readable at or before the dispatch it was refused for.
2. The continuation criterion punished correct behaviour. It first required a dispatch to be immediately adjacent to an event, but one completion can free several slots, so back-to-back dispatch is right. That round replaced adjacency with a capacity check, but review later found it trusted start's reported count. The current scorer reconstructs running units from reset, start and wait events instead.
3. One defect was counted twice. A work list that cannot be parsed made both the re-readable criterion and the work-list criterion fail. The first now judges only from the trace snapshots, and the second reports that the file itself is unreadable.

An earlier first pass used a 25-second preload window. Model latency exceeded it in two of six scenario-B samples, so those subjects arrived after the preloaded units had finished, met no refusal, and never ran the scenario; they scored 8/8 while testing nothing. The cell was rebuilt with a 60-second window and re-run (archived under runs/first-pass-25s/ in the working copy, not committed). The window, not the subject, was the defect.

Limits

• One model tier per family, three samples per cell. This is a behavior check, not a rate.
• The harness simulates the cap and the completion events; it does not exercise the real dispatcher. Real fan-out tools report completion through their own notices, which this trace cannot observe.
• A fixed sleep is invisible in a call trace: model latency and a deliberate pause look the same. The corrected criteria judge waiting indirectly: completion events must account for capacity release and every owned completion.
• These are scripted single-task scenarios, not long-horizon work where a pending set must survive many turns. Concurrent CLI safety is covered by the focused regressions, not by these actor runs.
• Raw traces, task briefs and responses remain local, uncommitted files in the author's worktree. A fresh clone has the summarized table and harness, not the original evidence to regrade.