nix-config-ryan4yin/agents/evals/git-delivery/runs/2026-10-10.md

Git delivery fixture run

Date: 2026-10-10

Purpose: compare the shortened git-delivery skill with the version before this round's edits on the fixed fixtures.

Inputs

• Fixtures: ../fixtures.json; grading: ../README.md.
• Baseline: git show 13f7c007:agents/skills/git-delivery/{SKILL.md,examples.md}.
• Current: agents/skills/git-delivery/ at the commit that adds this run, whose examples.md no longer contains the fixtures' changes.
• Model: gpt-6-luna through DeepSeek Harness 0.2.0-rc.2 subagents; one fresh agent and one sample per fixture and arm. The prompt allowed reading only the two skill files and fixtures.json; the agents' tool logs were not inspected.
• Counts: measured by a script (subject characters, body sentences and words without trailers, PR whitespace-separated words and paragraphs); the models' own counts are not used.

Results

Each line: baseline / current.

• F1 bump — Pass / Partial. Both subject-only. The current subject fix(worker-lib): bump to 1.2.4 drops the fix's purpose, which fits the ceiling.
• F2 restructure — Partial / Pass. The baseline PR is one paragraph, omits why the change spans several areas, and adds an AI-disclosure line the repository does not require. The current PR covers breadth, review start, and the shared guide in three paragraphs at 80 words.
• F3 state-clear — Pass / Fail. The baseline body names deletion, survival, and the drain precondition, with Refs: #87. The current body omits the drain precondition (it is only in the PR) and has no Refs trailer.
• F4 build-rule — Pass / Pass. Both subject-only, rationale in the PR.
• F5 commit-reference — Partial / Pass. The baseline body states the motive and leaves the unrecoverable deletions to the PR. The current body records both effects and cites the hash with its subject.
• F6 format-migration — Pass / Pass. Both keep the stored-format effect in the body and the rollout order, risk, verification, and status in the PR.

Each arm passes 4 of 6. The current skill separates PR review points (2–3 paragraphs in 5 of 6 outputs; the baseline wrote one paragraph every time) and fixes F2 and F5, but loses the subject purpose on F1 and the drain precondition on F3. With one sample per cell this is a direction, not a rate. Every output reported at least one wrong count, e.g. 69 PR words reported for 80 measured.

Measured counts

Subject chars; body sentences/words; PR words/paragraphs.

• Baseline: F1 54; 0/0; 41/1 · F2 35; 0/0; 69/1 · F3 38; 2/30; 66/1 · F4 36; 0/0; 46/1 · F5 48; 1/22; 49/1 · F6 46; 1/12; 62/1
• Current: F1 30; 0/0; 46/1 · F2 43; 0/0; 80/3 · F3 45; 1/14; 60/2 · F4 44; 0/0; 47/2 · F5 48; 1/14; 41/2 · F6 46; 1/11; 63/2

Current arm outputs

F1 bump

fix(worker-lib): bump to 1.2.4

Pin worker-lib to 1.2.4 to fix retry handling under intermittent connection resets. Revision bbb2222 passed a retry smoke test on Linux x86_64; macOS was not tested. The 35% reduction in failed jobs came from load testing revision aaa1111 before the final pin and was not remeasured.

F2 restructure

refactor(docs): group prose tables as lists

Convert prose tables to grouped lists and move duplicated activation guidance into the shared command guide. Keep mapping tables as tables and remove the padding lint guard.

The changes share the same table-padding problem across global guidance, repo skills, and eval instructions. Review the shared command guide first to confirm the local activation boundary remains clear.

Prettier previously rewrote 187 lines for three table-entry changes. Revision ccc3333 passed a semantic content comparison on the documentation; no runtime test was relevant.

F3 state-clear

fix(worker): clear stale state before startup

Every restart deletes cached state, including pending jobs; completed jobs remain in the database.

Every restart removes cached state before startup, including pending jobs; completed jobs remain in the database. Drain pending jobs before restarting. Removing the pre-start command prevents further deletion but cannot restore deleted jobs.

This works around upstream issue #87; it does not resolve it. Revision ddd4444 passed the stale-state restart test on Linux x86_64. No live host has been activated.

F4 build-rule

feat(build): add Python 3 rule for status.py

Add a Python 3 build rule for status.py alongside its Python 2 rule, so consumers can reuse it instead of defining their own. Keeping both rules together also simplifies upcoming automated build-file refactoring.

Revision eee5555 built the new rule on Linux x86_64; runtime behavior was not tested.

F5 commit-reference

fix(worker): retain pending jobs across restarts

Pending jobs survive restarts again; jobs deleted by the prior workaround cannot be recovered.

Refs: 1a2b3c4 (fix(worker): clear stale state before startup)

Removing the obsolete pre-start deletion preserves pending jobs across restarts. Stale state is now rejected by the worker library.

Previously deleted jobs cannot be recovered. Revision fff6666 passed the pending-job restart test on Linux x86_64; no live host has been activated.

F6 format-migration

fix(cache): store expiry timestamps in seconds

Old millisecond entries are invalidated on first read and repopulated automatically.

Cache expiry timestamps are now stored in seconds; readers invalidate old millisecond entries and repopulate the cache automatically.

Deploy readers that accept seconds before writers. Rolling back writers is safe while those readers remain deployed. Cache warming may increase database reads; no performance improvement was measured. Revision aaa7777 passed old-format invalidation and new-format round-trip tests on Linux x86_64. No production rollout has happened.

Baseline arm outputs

F1 bump

fix(worker-lib): handle intermittent connection resets

Pin worker-lib to 1.2.4 to fix retry handling under intermittent connection resets. Retry smoke test passed on Linux x86_64 at bbb2222; macOS was not tested. The 35% failed-job reduction was measured at aaa1111 before the final pin and was not repeated.

F2 restructure

docs: group prose tables into lists

Replace prose tables with grouped lists and remove the table-padding lint guard that caused Prettier to rewrite 187 lines for three changed entries. Move duplicated activation guidance into the shared command guide; mapping tables remain tables, and local activation remains user-run. Review the shared command guide first to verify activation boundaries. Semantic content comparison passed at ccc3333 on documentation; no runtime test was relevant. AI assistance: DeepSeek Harness (gpt-6-luna).

F3 state-clear

fix(worker): clear stale startup state

Every restart deletes cached state, including pending jobs; completed jobs remain in the database. Operators must drain pending jobs before restarting because removing the pre-start command cannot restore deleted jobs.

Refs: #87

Add a pre-start cleanup to work around the stale-state startup failure reported in #87. Every restart deletes cached state, including pending jobs, so operators must drain pending jobs first; completed jobs remain in the database, and removing the command cannot restore deleted jobs. The stale-state restart test passed on Linux x86_64 (revision ddd4444); no live host was activated. This workaround does not resolve the upstream issue.

F4 build-rule

feat(build): add Python3 status rule

Add a local Python3 build rule for status.py beside the existing Python2 rule, so consumers can reuse it instead of defining their own and upcoming automated build-file refactoring can keep related rules together. Revision eee5555 built the rule on Linux x86_64; runtime behavior was not tested.

F5 commit-reference

fix(worker): retain pending jobs across restarts

Remove the pre-start deletion that discarded pending jobs; stale state is now rejected by the worker library, so the workaround is obsolete.

Refs: 1a2b3c4 fix(worker): clear stale state before startup

Remove the pre-start state deletion so pending jobs survive worker restarts; previously deleted jobs cannot be recovered by this fix. The worker library now rejects stale state, making the deletion workaround obsolete. The pending-job restart test passed on Linux x86_64 at revision fff6666; no live host has been activated.

F6 format-migration

fix(cache): store expiry timestamps in seconds

Old millisecond cache entries are invalidated on first read and repopulate automatically.

Cache expiry timestamps now use seconds; readers invalidate old millisecond entries and repopulate the cache automatically. Deploy readers that accept seconds before writers; writer rollback is safe while those readers remain deployed. Cache warming may increase database reads. No performance improvement was measured, and no production rollout has happened. Tests passed on revision aaa7777: old-format invalidation and new-format round-trip on Linux x86_64.