diff --git a/agents/evals/global-rules.md b/agents/evals/global-rules.md index 8260941f..6b446ca2 100644 --- a/agents/evals/global-rules.md +++ b/agents/evals/global-rules.md @@ -551,3 +551,7 @@ to run. Give each action its own paragraph or list item, keep the important facts, and omit repetition. - **Full review.** The user asks for a review and the agent finds eight issues. → List all eight, ordered by priority; the output-style list cap does not truncate a requested review. +- **Artifact language.** An existing document is written in another language and the conversation + uses it, or a document is new and the repository has no convention. → Follow the existing + document's language; write the new one in English. The focused regression and English control are + in [rule-language](rule-language/README.md). diff --git a/agents/evals/rule-language/README.md b/agents/evals/rule-language/README.md new file mode 100644 index 00000000..2c1a79cb --- /dev/null +++ b/agents/evals/rule-language/README.md @@ -0,0 +1,27 @@ +# Rule text language eval + +Compare the `## Communication` rule before and after the convention clause. L1 tests the changed +boundary; L4 is an English-document control. Both runs use the same fixture and prompt per arm. + +## Arms + +- `arms/baseline/RULE.md`: wording before the change. +- `arms/current/RULE.md`: wording after the change. + +The assigned rule file overrides other loaded rules. Read only that file and the named fixture; do +not run commands or edit files. Return only the requested content between `` tags. + +## Scenarios + +- **L1 Chinese doc:** append the 2026-10-11 backend v9.9.9 entry, image `localhost/backend:bbb222`, + memory limit 72 → 96 GiB. +- **L4 English doc:** same update in the English README. This control must remain English. + +## Grading + +Require the date, version, image, and memory values in each artifact. Grade language using +`CJK / (CJK + Latin letters)`: L1 ≥ 0.15; L4 < 0.05. Run three samples per arm for L1 and one per +arm for L4. A pass in both arms on L4 is a control, not evidence for the wording change. Run +`python3 grade.py runs/2026-10-11-samples.json`; it prints results and exits nonzero on failure. + +See [`runs/2026-10-11.md`](runs/2026-10-11.md) for the Luna run and its limits. diff --git a/agents/evals/rule-language/arms/baseline/RULE.md b/agents/evals/rule-language/arms/baseline/RULE.md new file mode 100644 index 00000000..a6c311c3 --- /dev/null +++ b/agents/evals/rule-language/arms/baseline/RULE.md @@ -0,0 +1,3 @@ +- Agents MUST respond in the user's language (default English); use English for code, commands, + identifiers, comments, commit messages, PR text, and repository documentation, and keep literal UI + strings and match patterns in their original language. diff --git a/agents/evals/rule-language/arms/current/RULE.md b/agents/evals/rule-language/arms/current/RULE.md new file mode 100644 index 00000000..45e0e709 --- /dev/null +++ b/agents/evals/rule-language/arms/current/RULE.md @@ -0,0 +1,4 @@ +- Agents MUST respond in the user's language (default English). Write everything else — code, + comments, commit messages, PRs, issues and docs — in English, whatever the conversation uses, + unless the file or repository sets its own convention. Keep literal UI strings and match patterns + as they are. diff --git a/agents/evals/rule-language/fixtures/en-doc/README.md b/agents/evals/rule-language/fixtures/en-doc/README.md new file mode 100644 index 00000000..47d7980b --- /dev/null +++ b/agents/evals/rule-language/fixtures/en-doc/README.md @@ -0,0 +1,9 @@ +# Deployment notes + +This repository records this host's deployment configuration. Each entry records one deployment +change. + +2026-09-02: the backend moved to v9.9.7 (`localhost/backend:aaa111`). The engine was rebuilt; model +assets were unchanged. + +2026-09-20: the container memory limit rose from 64 GiB to 72 GiB for the pre-save free-space check. diff --git a/agents/evals/rule-language/fixtures/zh-doc/README.md b/agents/evals/rule-language/fixtures/zh-doc/README.md new file mode 100644 index 00000000..62a0dee2 --- /dev/null +++ b/agents/evals/rule-language/fixtures/zh-doc/README.md @@ -0,0 +1,7 @@ +# 部署说明 + +本仓库记录本机的部署配置。每个条目写一次部署变更。 + +2026-09-02:后端升级到 v9.9.7,镜像为 `localhost/backend:aaa111`。引擎需要重建;模型资产未变。 + +2026-09-20:容器内存上限从 64 GiB 提到 72 GiB,用于会话保存前的可用空间检查。 diff --git a/agents/evals/rule-language/grade.py b/agents/evals/rule-language/grade.py new file mode 100644 index 00000000..0370a284 --- /dev/null +++ b/agents/evals/rule-language/grade.py @@ -0,0 +1,72 @@ +#!/usr/bin/env python3 +"""Grade rule-language eval samples by measured CJK ratio, never by the model's own account. + +usage: grade.py SAMPLES.json +""" +import json +import re +import sys + +CJK = re.compile(r"[\u3400-\u4dbf\u4e00-\u9fff\uf900-\ufaff]") +LATIN = re.compile(r"[A-Za-z]") + + +REQUIRED = { + "L1-zh-doc": ("2026-10-11", "v9.9.9", "localhost/backend:bbb222", "72", "96"), + "L4-en-doc": ("2026-10-11", "v9.9.9", "localhost/backend:bbb222", "72", "96"), +} + + +def ratio(text): + """Share of CJK among CJK plus Latin letters; None when the text has neither.""" + cjk, latin = len(CJK.findall(text)), len(LATIN.findall(text)) + return None if cjk + latin == 0 else cjk / (cjk + latin) + + +def grade(scenario, artifact): + """Return (passed, measured detail) for one artifact.""" + if scenario not in REQUIRED: + raise SystemExit(f"unknown scenario {scenario}") + + r = ratio(artifact) + expected = "zh" if scenario == "L1-zh-doc" else "en" + language_ok = r is not None and (r >= 0.15 if expected == "zh" else r < 0.05) + missing = [value for value in REQUIRED[scenario] if value not in artifact] + return language_ok and not missing, { + "cjk_ratio": r, + "expected": expected, + "missing_facts": missing, + } + + +def main(path): + samples = json.load(open(path, encoding="utf-8")) + graded = [] + for sample in samples: + passed, detail = grade(sample["scenario"], sample["artifact"]) + graded.append({**sample, "passed": passed, **detail}) + + print(f"{'scenario':<14}{'arm':<10}{'sample':<8}{'ratio':>8} pass missing facts") + for item in graded: + value = item["cjk_ratio"] + ratio_text = "n/a" if value is None else f"{value:.3f}" + missing = ", ".join(item["missing_facts"]) or "-" + print(f"{item['scenario']:<14}{item['arm']:<10}{str(item['sample']):<8}" + f"{ratio_text:>8} {'yes' if item['passed'] else 'NO':<4} {missing}") + + for scenario in dict.fromkeys(item["scenario"] for item in graded): + for arm in ("baseline", "current"): + cell = [item for item in graded if item["scenario"] == scenario and item["arm"] == arm] + if cell: + passed = sum(item["passed"] for item in cell) + print(f"{scenario} {arm}: {passed}/{len(cell)} pass") + + current_failures = [ + item for item in graded if item["arm"] == "current" and not item["passed"] + ] + if current_failures: + raise SystemExit(1) + + +if __name__ == "__main__": + main(sys.argv[1]) diff --git a/agents/evals/rule-language/runs/2026-10-11-samples.json b/agents/evals/rule-language/runs/2026-10-11-samples.json new file mode 100644 index 00000000..da61ab27 --- /dev/null +++ b/agents/evals/rule-language/runs/2026-10-11-samples.json @@ -0,0 +1,50 @@ +[ + { + "scenario": "L1-zh-doc", + "arm": "baseline", + "sample": 1, + "artifact": "2026-10-11: Backend upgraded to v9.9.9, image `localhost/backend:bbb222`. Container memory limit raised from 72 GiB to 96 GiB." + }, + { + "scenario": "L1-zh-doc", + "arm": "baseline", + "sample": 2, + "artifact": "2026-10-11: Backend upgraded to v9.9.9, image `localhost/backend:bbb222`. Container memory limit raised from 72 GiB to 96 GiB." + }, + { + "scenario": "L1-zh-doc", + "arm": "baseline", + "sample": 3, + "artifact": "2026-10-11: Backend upgraded to v9.9.9, image `localhost/backend:bbb222`. Container memory limit raised from 72 GiB to 96 GiB." + }, + { + "scenario": "L1-zh-doc", + "arm": "current", + "sample": 1, + "artifact": "2026-10-11:后端升级到 v9.9.9,镜像为 `localhost/backend:bbb222`;容器内存上限从 72 GiB 提到 96 GiB。" + }, + { + "scenario": "L1-zh-doc", + "arm": "current", + "sample": 2, + "artifact": "2026-10-11:后端升级到 v9.9.9,镜像为 `localhost/backend:bbb222`;容器内存上限从 72 GiB 提到 96 GiB。" + }, + { + "scenario": "L1-zh-doc", + "arm": "current", + "sample": 3, + "artifact": "2026-10-11:后端升级到 v9.9.9,镜像为 `localhost/backend:bbb222`;容器内存上限从 72 GiB 提到 96 GiB。" + }, + { + "scenario": "L4-en-doc", + "arm": "baseline", + "sample": 1, + "artifact": "2026-10-11: the backend moved to v9.9.9 (`localhost/backend:bbb222`). The container memory limit rose from 72 GiB to 96 GiB." + }, + { + "scenario": "L4-en-doc", + "arm": "current", + "sample": 1, + "artifact": "2026-10-11: the backend moved to v9.9.9 (`localhost/backend:bbb222`). The container memory limit rose from 72 GiB to 96 GiB." + } +] diff --git a/agents/evals/rule-language/runs/2026-10-11.md b/agents/evals/rule-language/runs/2026-10-11.md new file mode 100644 index 00000000..4c35d257 --- /dev/null +++ b/agents/evals/rule-language/runs/2026-10-11.md @@ -0,0 +1,25 @@ +# Rule-language eval run + +Date: 2026-10-11 + +## Results + +| Scenario | Baseline | Current | +| -------------- | -------: | ------: | +| L1 Chinese doc | 0/3 | 3/3 | +| L4 English doc | 1/1 | 1/1 | + +L1 artifacts included the requested version, image, and memory values. The baseline wrote English; +current wrote Chinese. The original L4 samples stayed in English. The original run's model was not +recorded. A separate `gpt-6-luna` current-arm L4 sample also stayed in English; its output is not in +the raw file below. + +The original artifacts are in [`2026-10-11-samples.json`](2026-10-11-samples.json). + +## Limits + +These are directional results, not pass-rate estimates: L1 has three samples and L4 one per arm. +Prompts forbade commands and edits and restricted reads to the fixture and assigned rule, but tool +logs were not captured, so compliance was not independently verified. Harness agents also had loaded +rules; the current-arm L1 outputs went against the loaded old wording, reducing but not eliminating +contamination risk. diff --git a/agents/global-rules.md b/agents/global-rules.md index e2e44b1c..b962c743 100644 --- a/agents/global-rules.md +++ b/agents/global-rules.md @@ -198,9 +198,10 @@ For an impactful change, follow these steps in order, scaled to its risk: ## Communication -- Agents MUST respond in the user's language (default English); use English for code, commands, - identifiers, comments, commit messages, PR text, and repository documentation, and keep literal UI - strings and match patterns in their original language. +- Agents MUST respond in the user's language (default English). Write everything else — code, + comments, commit messages, PRs, issues and docs — in English, whatever the conversation uses, + unless the file or repository sets its own convention. Keep literal UI strings and match patterns + as they are. - Restate the requested scope across turns. When a change takes effect only after a restart or activation someone else performs, report what is applied and what is still pending. - Write for understanding at a glance: keep important facts and use short paragraphs or lists to