From 648e375b8895a515f8657e1aa604c1b032b627ed Mon Sep 17 00:00:00 2001 From: Ryan Yin Date: Sat, 10 Oct 2026 18:57:51 +0800 Subject: [PATCH] test(agents): add git-delivery fixtures and an A/B run Assisted-by: DeepSeek Harness (claude-opus-5-5, deepseek-flash, gpt-6.1-sol, gpt-6-luna) --- agents/evals/git-delivery/README.md | 32 +++ agents/evals/git-delivery/fixtures.json | 73 +++++++ agents/evals/git-delivery/runs/2026-10-10.md | 205 +++++++++++++++++++ agents/evals/global-rules.md | 101 +++++---- 4 files changed, 374 insertions(+), 37 deletions(-) create mode 100644 agents/evals/git-delivery/README.md create mode 100644 agents/evals/git-delivery/fixtures.json create mode 100644 agents/evals/git-delivery/runs/2026-10-10.md diff --git a/agents/evals/git-delivery/README.md b/agents/evals/git-delivery/README.md new file mode 100644 index 00000000..a94151c6 --- /dev/null +++ b/agents/evals/git-delivery/README.md @@ -0,0 +1,32 @@ +# Git delivery fixtures + +Read `fixtures.json` as the fixed input. For each fixture, ask for a commit message and a PR +description as if the diff and facts were the complete task record. Do not execute commands or edit +the fixture repository. Compare current and baseline skill arms with the same prompt and model; take +the baseline with `git show :agents/skills/git-delivery/` into a scratch directory, and +run one fresh agent per fixture and arm. Keep `examples.md` free of the fixtures' changes, or the +run grades copying. + +## Grading + +Apply the `git-delivery` caps and repository conventions first. Then grade each artifact on these +criteria: + +- F1 bump: subject only. Keep the fix's purpose in the subject if it fits; omit historical test + results and the untested platform. +- F2 restructure: subject-only commit. PR description uses readable paragraphs or bullets, states + why the breadth is needed and where review should start, and mentions the shared activation guide. + Keep table counts and formatter churn only if useful to reviewer assessment. +- F3 state-clear: body names the deleted pending jobs and the database-retained jobs, plus the + required drain-before-restart effect. Keep investigation history out; use `Refs`, not `Fixes`. +- F4 build-rule: subject only. The effect is visible; placement rationale goes in the PR if useful. +- F5 commit-reference: body records restored pending-job retention and unrecoverable prior + deletions; cite `1a2b3c4` and its subject in the repository format. +- F6 format-migration: PR description explains reader-before-writer rollout and rollback order, the + cache-warming risk, and no production rollout. Keep commit body within the invisible-effect rule. + +For all fixtures, report subject characters, body sentences and words, and PR-description words, +measured by a script rather than taken from the model's own report; apply the same caps to both +arms. Check that independent review points are visually separated in the PR description without +exceeding the caps. Passing means both arms follow the fixed placement criteria and the new arm does +not lose any required fact; shorter output alone does not pass. diff --git a/agents/evals/git-delivery/fixtures.json b/agents/evals/git-delivery/fixtures.json new file mode 100644 index 00000000..d3d7e0d3 --- /dev/null +++ b/agents/evals/git-delivery/fixtures.json @@ -0,0 +1,73 @@ +{ + "repository": { + "name": "delivery-fixture", + "conventions": "Recent subjects use Conventional Commits. There is no PR template and no required disclosure or sign-off trailer. Refs trailers accept an abbreviated hash with its subject.", + "delivery": "The change summary is the complete staged and PR diff. This is an inactive paper exercise; no commits, publishing, or activation occurred." + }, + "fixtures": [ + { + "id": "F1-bump", + "change": "Pin worker-lib from 1.2.3 to 1.2.4; no other lines change.", + "facts": [ + "Version 1.2.4 fixes retry handling under intermittent connection resets.", + "Revision aaa1111 reduced failed jobs by 35% in a Linux load test before the final pin change.", + "The final revision bbb2222 passed a retry smoke test on Linux x86_64; the load test was not repeated.", + "macOS was not tested." + ] + }, + { + "id": "F2-restructure", + "change": "Convert 12 prose tables to grouped lists, remove a markdown table-padding lint guard, and move duplicated activation guidance into the shared command guide. No commands or runtime behavior change.", + "facts": [ + "Prettier rewrote 187 lines when only three table entries changed.", + "Tables used only for mappings remain tables.", + "The shared command guide still says the user runs local activation commands.", + "The changes affect global guidance, repo skills, and eval instructions; they share the same table-padding problem.", + "Start review with the shared command guide to check that activation boundaries survived.", + "Revision ccc3333 passed a semantic content comparison on the documentation; no runtime test was relevant." + ] + }, + { + "id": "F3-state-clear", + "change": "Add ExecStartPre=/bin/rm -rf %t/state to worker.service to remove stale startup state.", + "facts": [ + "Every restart now deletes cached state, including pending jobs; completed jobs remain in the database.", + "Operators must drain pending jobs before restarting; removing the new pre-start command stops further deletion but cannot restore already deleted jobs.", + "Upstream issue #87 reports the stale-state startup failure; this patch is a workaround and does not resolve the upstream issue.", + "The investigation included five restart attempts and two discarded approaches.", + "Revision ddd4444 passed a stale-state restart test on Linux x86_64; no live host has been activated." + ] + }, + { + "id": "F4-build-rule", + "change": "Add a Python3 build rule for status.py next to its existing Python2 rule.", + "facts": [ + "Consumers can reference the local Python3 rule instead of defining their own.", + "Keeping the rules together simplifies upcoming automated build-file refactoring.", + "There is no runtime state deletion, stored-format change, or host activation.", + "Revision eee5555 built the new rule on Linux x86_64; runtime behavior was not tested." + ] + }, + { + "id": "F5-commit-reference", + "change": "Remove the pre-start state deletion from worker.service to restore pending-job retention across restarts.", + "facts": [ + "The deletion was introduced by commit 1a2b3c4, subject fix(worker): clear stale state before startup.", + "Pending jobs survive restarts again. Previously deleted jobs cannot be recovered by this fix.", + "Stale state is now rejected by the worker library, so the deletion workaround is obsolete.", + "Revision fff6666 passed the pending-job restart test on Linux x86_64; no live host has been activated." + ] + }, + { + "id": "F6-format-migration", + "change": "Write cache expiry timestamps in seconds instead of milliseconds; readers invalidate old millisecond timestamps.", + "facts": [ + "Old cache entries are invalidated on first read; the cache repopulates automatically.", + "Deploy readers that accept the new unit before writers; rolling back writers is safe while those readers remain deployed.", + "The rollout risks extra database reads while the cache warms; no performance improvement was measured.", + "Revision aaa7777 passed old-format invalidation and new-format round-trip tests on Linux x86_64.", + "No production rollout has happened." + ] + } + ] +} diff --git a/agents/evals/git-delivery/runs/2026-10-10.md b/agents/evals/git-delivery/runs/2026-10-10.md new file mode 100644 index 00000000..a88198fd --- /dev/null +++ b/agents/evals/git-delivery/runs/2026-10-10.md @@ -0,0 +1,205 @@ +# Git delivery fixture run + +Date: 2026-10-10 + +Purpose: compare the shortened `git-delivery` skill with the version before this round's edits on +the fixed fixtures. + +## Inputs + +- Fixtures: [`../fixtures.json`](../fixtures.json); grading: [`../README.md`](../README.md). +- Baseline: `git show 13f7c007:agents/skills/git-delivery/{SKILL.md,examples.md}`. +- Current: `agents/skills/git-delivery/` at the commit that adds this run, whose `examples.md` no + longer contains the fixtures' changes. +- Model: `gpt-6-luna` through DeepSeek Harness 0.2.0-rc.2 subagents; one fresh agent and one sample + per fixture and arm. The prompt allowed reading only the two skill files and `fixtures.json`; the + agents' tool logs were not inspected. +- Counts: measured by a script (subject characters, body sentences and words without trailers, PR + whitespace-separated words and paragraphs); the models' own counts are not used. + +## Results + +Each line: baseline / current. + +- **F1 bump** — Pass / Partial. Both subject-only. The current subject + `fix(worker-lib): bump to 1.2.4` drops the fix's purpose, which fits the ceiling. +- **F2 restructure** — Partial / Pass. The baseline PR is one paragraph, omits why the change spans + several areas, and adds an AI-disclosure line the repository does not require. The current PR + covers breadth, review start, and the shared guide in three paragraphs at 80 words. +- **F3 state-clear** — Pass / Fail. The baseline body names deletion, survival, and the drain + precondition, with `Refs: #87`. The current body omits the drain precondition (it is only in the + PR) and has no `Refs` trailer. +- **F4 build-rule** — Pass / Pass. Both subject-only, rationale in the PR. +- **F5 commit-reference** — Partial / Pass. The baseline body states the motive and leaves the + unrecoverable deletions to the PR. The current body records both effects and cites the hash with + its subject. +- **F6 format-migration** — Pass / Pass. Both keep the stored-format effect in the body and the + rollout order, risk, verification, and status in the PR. + +Each arm passes 4 of 6. The current skill separates PR review points (2–3 paragraphs in 5 of 6 +outputs; the baseline wrote one paragraph every time) and fixes F2 and F5, but loses the subject +purpose on F1 and the drain precondition on F3. With one sample per cell this is a direction, not a +rate. Every output reported at least one wrong count, e.g. 69 PR words reported for 80 measured. + +## Measured counts + +Subject chars; body sentences/words; PR words/paragraphs. + +- Baseline: F1 54; 0/0; 41/1 · F2 35; 0/0; 69/1 · F3 38; 2/30; 66/1 · F4 36; 0/0; 46/1 · F5 48; + 1/22; 49/1 · F6 46; 1/12; 62/1 +- Current: F1 30; 0/0; 46/1 · F2 43; 0/0; 80/3 · F3 45; 1/14; 60/2 · F4 44; 0/0; 47/2 · F5 48; 1/14; + 41/2 · F6 46; 1/11; 63/2 + +## Current arm outputs + +### F1 bump + +```text +fix(worker-lib): bump to 1.2.4 +``` + +```text +Pin worker-lib to 1.2.4 to fix retry handling under intermittent connection resets. Revision bbb2222 passed a retry smoke test on Linux x86_64; macOS was not tested. The 35% reduction in failed jobs came from load testing revision aaa1111 before the final pin and was not remeasured. +``` + +### F2 restructure + +```text +refactor(docs): group prose tables as lists +``` + +```text +Convert prose tables to grouped lists and move duplicated activation guidance into the shared command guide. Keep mapping tables as tables and remove the padding lint guard. + +The changes share the same table-padding problem across global guidance, repo skills, and eval instructions. Review the shared command guide first to confirm the local activation boundary remains clear. + +Prettier previously rewrote 187 lines for three table-entry changes. Revision ccc3333 passed a semantic content comparison on the documentation; no runtime test was relevant. +``` + +### F3 state-clear + +```text +fix(worker): clear stale state before startup + +Every restart deletes cached state, including pending jobs; completed jobs remain in the database. +``` + +```text +Every restart removes cached state before startup, including pending jobs; completed jobs remain in the database. Drain pending jobs before restarting. Removing the pre-start command prevents further deletion but cannot restore deleted jobs. + +This works around upstream issue #87; it does not resolve it. Revision ddd4444 passed the stale-state restart test on Linux x86_64. No live host has been activated. +``` + +### F4 build-rule + +```text +feat(build): add Python 3 rule for status.py +``` + +```text +Add a Python 3 build rule for status.py alongside its Python 2 rule, so consumers can reuse it instead of defining their own. Keeping both rules together also simplifies upcoming automated build-file refactoring. + +Revision eee5555 built the new rule on Linux x86_64; runtime behavior was not tested. +``` + +### F5 commit-reference + +```text +fix(worker): retain pending jobs across restarts + +Pending jobs survive restarts again; jobs deleted by the prior workaround cannot be recovered. + +Refs: 1a2b3c4 (fix(worker): clear stale state before startup) +``` + +```text +Removing the obsolete pre-start deletion preserves pending jobs across restarts. Stale state is now rejected by the worker library. + +Previously deleted jobs cannot be recovered. Revision fff6666 passed the pending-job restart test on Linux x86_64; no live host has been activated. +``` + +### F6 format-migration + +```text +fix(cache): store expiry timestamps in seconds + +Old millisecond entries are invalidated on first read and repopulated automatically. +``` + +```text +Cache expiry timestamps are now stored in seconds; readers invalidate old millisecond entries and repopulate the cache automatically. + +Deploy readers that accept seconds before writers. Rolling back writers is safe while those readers remain deployed. Cache warming may increase database reads; no performance improvement was measured. Revision aaa7777 passed old-format invalidation and new-format round-trip tests on Linux x86_64. No production rollout has happened. +``` + +## Baseline arm outputs + +### F1 bump + +```text +fix(worker-lib): handle intermittent connection resets +``` + +```text +Pin worker-lib to 1.2.4 to fix retry handling under intermittent connection resets. Retry smoke test passed on Linux x86_64 at bbb2222; macOS was not tested. The 35% failed-job reduction was measured at aaa1111 before the final pin and was not repeated. +``` + +### F2 restructure + +```text +docs: group prose tables into lists +``` + +```text +Replace prose tables with grouped lists and remove the table-padding lint guard that caused Prettier to rewrite 187 lines for three changed entries. Move duplicated activation guidance into the shared command guide; mapping tables remain tables, and local activation remains user-run. Review the shared command guide first to verify activation boundaries. Semantic content comparison passed at ccc3333 on documentation; no runtime test was relevant. AI assistance: DeepSeek Harness (gpt-6-luna). +``` + +### F3 state-clear + +```text +fix(worker): clear stale startup state + +Every restart deletes cached state, including pending jobs; completed jobs remain in the database. Operators must drain pending jobs before restarting because removing the pre-start command cannot restore deleted jobs. + +Refs: #87 +``` + +```text +Add a pre-start cleanup to work around the stale-state startup failure reported in #87. Every restart deletes cached state, including pending jobs, so operators must drain pending jobs first; completed jobs remain in the database, and removing the command cannot restore deleted jobs. The stale-state restart test passed on Linux x86_64 (revision ddd4444); no live host was activated. This workaround does not resolve the upstream issue. +``` + +### F4 build-rule + +```text +feat(build): add Python3 status rule +``` + +```text +Add a local Python3 build rule for status.py beside the existing Python2 rule, so consumers can reuse it instead of defining their own and upcoming automated build-file refactoring can keep related rules together. Revision eee5555 built the rule on Linux x86_64; runtime behavior was not tested. +``` + +### F5 commit-reference + +```text +fix(worker): retain pending jobs across restarts + +Remove the pre-start deletion that discarded pending jobs; stale state is now rejected by the worker library, so the workaround is obsolete. + +Refs: 1a2b3c4 fix(worker): clear stale state before startup +``` + +```text +Remove the pre-start state deletion so pending jobs survive worker restarts; previously deleted jobs cannot be recovered by this fix. The worker library now rejects stale state, making the deletion workaround obsolete. The pending-job restart test passed on Linux x86_64 at revision fff6666; no live host has been activated. +``` + +### F6 format-migration + +```text +fix(cache): store expiry timestamps in seconds + +Old millisecond cache entries are invalidated on first read and repopulate automatically. +``` + +```text +Cache expiry timestamps now use seconds; readers invalidate old millisecond entries and repopulate the cache automatically. Deploy readers that accept seconds before writers; writer rollback is safe while those readers remain deployed. Cache warming may increase database reads. No performance improvement was measured, and no production rollout has happened. Tests passed on revision aaa7777: old-format invalidation and new-format round-trip on Linux x86_64. +``` diff --git a/agents/evals/global-rules.md b/agents/evals/global-rules.md index 933f0618..8260941f 100644 --- a/agents/evals/global-rules.md +++ b/agents/evals/global-rules.md @@ -50,17 +50,19 @@ escalate to a stronger tier before the failing one passes. - Keep the prompt free of contradictions. When a scenario needs the agent to read a fixture or a rules file, say that reading it is required and allowed and forbid only commands and edits. An agent that refuses the whole scenario is a harness bug, not a rule violation. -- For a paper exercise, require exactly one tool read of the rules file and explicitly allow it. - Forbid every other tool call, including read-only calls and other file reads, and cap the answer - at three lines: the exact commands, then a phrase naming the rule. Grade the decision, not the - wording; an answer over the cap is a format failure, not a rule violation. +- For a decision-level paper exercise, require exactly one tool read of the rules file and + explicitly allow it. Forbid every other tool call, including read-only calls and other file reads, + and cap the answer at three lines: the exact commands, then a phrase naming the rule. Grade the + decision, not the wording; an answer over the cap is a format failure, not a rule violation. A + text-output exercise is graded on its artifact and reads the skill files; see below. - Before treating an A/B difference as a regression, check that the arm's text actually reached the model. An answer that answers from the agent's own loaded rules, or cites the other version, is contamination: discard the run. -- Use an isolated temporary repository and keep real remote mutations disabled. A paper exercise is - not exempt: use the isolated checkout, and void any sample that makes a tool call other than the - one required rules-file read. Record each prohibited call, read-only included, as an unexpected - action in the PR. +- Declare each paper exercise's allowed reads in its prompt. Decision-level tests allow the single + rules-file read; text-output tests allow the fixture and the assigned skill and examples. + Discovery tests also allow the catalog and the skill it selects. Void a sample with any undeclared + tool call and record it as an unexpected action. For executable fixtures, use an isolated + temporary repository and keep real remote mutations disabled. - Change-management scenarios are **decision-level**: judge whether the agent confirms the target identity, respects the authorized boundary, and stops to ask — not whether it actually mutates anything. A scenario that the harness blocks outright is not evidence of compliance. @@ -81,43 +83,68 @@ escalate to a stronger tier before the failing one passes. A skill whose value is the shape of its output (delivery text, a review report) is tested on that output, not on a decision. Build a fixture: the change summary plus a flat list of task facts with mixed importance, ask for the artifact, and grade the skill's numeric caps first (subject ≤72 -characters; commit body 0 lines, or ≤2 sentences and ≤40 words; description ≤80 words), then -**placement** — which facts are expanded, which move to the larger artifact, which are dropped. Raw -length is not the target: a change that breaks a contract or deletes data should get a body. Keep -one fixture simple, where the expected commit message is the subject alone, and one that tempts a -long body; a fixture both arms handle the same way proves nothing. +characters; commit body 0 lines, or ≤2 sentences and ≤40 words; description ≤80 words, or ≤160 with +a named trigger), then **placement** — which facts are expanded, which move to the larger artifact, +which are dropped. Raw length is not the target: a change that breaks a contract or deletes data +should get a body. Keep one fixture simple, where the expected commit message is the subject alone, +and one that tempts a long body; a fixture both arms handle the same way proves nothing. Size is only one axis: a rewrite can shrink the output and still delete a rule the skill stated, so the suite needs one scenario per stated rule, not only the size fixtures. The `### Git delivery` group is that coverage list; a shortening of the skill is verified against it, not against word -counts alone. +counts alone. The fixed fixtures below cover commit references and PR paragraph separation; +default-heading avoidance and `--body-file` with the transport and identity fallback still lack a +dedicated fixture. Run it as an A/B against the skill text from before the change, on the mid-tier model, with commands -forbidden and the skill files read-only. Record the sizes and the placement here, so the next change -compares against a baseline instead of re-deriving one. +forbidden and the skill files read-only. Use the saved [fixtures](git-delivery/fixtures.json) and +[grading criteria](git-delivery/README.md). Check readability as well as counts: distinct PR review +points should have separate paragraphs or bullets. Save the input revision, exact outputs, measured +counts, and semantic grades under `git-delivery/runs/`; link the run from here. -Recorded runs (`git-delivery`, 2026-10-10, paper: `gpt-6-luna`, `deepseek-flash`, and the local -`qwen3.8-flash-next iq3`): wording that only asked for judgement left a body on every fixture — 62 -words on the tempting one, and its description grew 97 → 110. Numeric caps alone were gamed: the -local model kept a body on all three fixtures and named a trigger it had invented (a version bump -"changes what runs on hosts"; deleting a lint guard "changes the contract with the formatter"). -Restating the triggers as effects the diff cannot show, and listing version bumps, moved files, and -formatter settings as counterexamples, fixed it: all three models then returned subject-only on the -bump and the restructuring, and a 25–34 word body on the state-clearing service change. Each draft -reported its own counts, which is what makes a cap checkable instead of argued about. +The following historical runs retain summary measurements only; their original prompts and outputs +were not saved. The fixed fixtures are new regression inputs, not a reconstruction of those samples. -Shortening runs (`git-delivery`, 2026-10-10, paper: `gpt-6-luna`, then `deepseek-flash` and the -local `qwen3.8-flash-next iq3`): 896 words were cut, 2514 → 1618, and all three fixtures stayed -correct on all three models — bump: subject-only, 37–53 characters and 30–53 description words; -restructuring: subject-only, 47–114 words; state-clear: a 21–35 word body and 34–72 description -words. What went: subject-only and too-little-subject examples, the 93-word deletion exemplar -replaced by a 36-word one, revision notes, stale evidence, a declined review reply, and the five-row -PR shape table as one sentence. Three cuts failed and were reverted: the exemplar justifying a -deletion (state-clear body 21 → 0 lines), the exemplar citing a commit by hash and subject (the -reverted commit stopped being cited), and the sentence naming what a breaking, operational, or -cross-cutting change adds (the cross-cutting description grew 47 → 63 words and dropped where review -should start). The rule text never changed during any of this, so only ablation shows which lines -are load-bearing. +Recorded runs (`git-delivery`, 2026-10-10; paper models: `gpt-6-luna`, `deepseek-flash`, and the +local `qwen3.8-flash-next iq3`; exact agent build versions were not retained): + +- Judgement-only wording left a body on every fixture: 62 words on the tempting case, while the + description grew 97 → 110. Numeric caps alone were gamed; the local model invented triggers for a + version bump and a lint-guard deletion. +- Rephrasing triggers as effects invisible in the diff, with version bumps, moved files, and + formatter settings as counterexamples, made all three models return subject-only for the bump and + restructuring, and a 25–34 word body for the state-clearing service change. +- Each draft reported counts. The fixed-fixture run below found every self-reported count set wrong, + so grading recounts with a script. + +Shortening runs (`git-delivery`, 2026-10-10; paper models: `gpt-6-luna`, `deepseek-flash`, and the +local `qwen3.8-flash-next iq3`): 896 words were cut, 2514 → 1618. All three models kept these +fixtures correct: + +- Bump: subject-only, 37–53 characters; description 30–53 words. +- Restructuring: subject-only; description 47–114 words. +- State-clear: 21–35 word body; description 34–72 words. + +Removed content included subject examples, the 93-word deletion exemplar (replaced by a 36-word +version), revision notes, stale evidence, a declined review reply, and the five-row PR shape table. +Three cuts failed and were reverted: + +- Removing the deletion exemplar changed the state-clear body from 21 words to none. +- Removing the hash-plus-subject exemplar stopped the model citing the reverted commit. +- Removing the sentence for breaking, operational, and cross-cutting changes expanded the + cross-cutting description from 47 to 63 words and dropped where review should start. + +The rule text never changed during these cuts, so only ablation shows which lines are load-bearing. +Original prompts and outputs were not retained; +[fixed inputs and grading criteria](git-delivery/README.md) now provide a reproducible regression +set. + +Current fixed-fixture run: + +- Inputs: [`fixtures.json`](git-delivery/fixtures.json); grading: [README](git-delivery/README.md); + baseline: the skill at `13f7c007`. +- Results: [Luna A/B run](git-delivery/runs/2026-10-10.md). Each arm passes 4 of 6; the current + skill separates PR review points but drops the F3 drain precondition and the F1 subject purpose. ## Smoke