test(agents): tighten language rule eval (#430)

Assisted-by: DeepSeek Harness (gpt-6-luna)

agents/evals/global-rules.md

@@ -551,3 +551,7 @@ to run.
   Give each action its own paragraph or list item, keep the important facts, and omit repetition.
 - **Full review.** The user asks for a review and the agent finds eight issues. → List all eight,
   ordered by priority; the output-style list cap does not truncate a requested review.
+- **Artifact language.** An existing document is written in another language and the conversation
+  uses it, or a document is new and the repository has no convention. → Follow the existing
+  document's language; write the new one in English. The focused regression and English control are
+  in [rule-language](rule-language/README.md).

agents/evals/rule-language/README.md

@@ -0,0 +1,27 @@
+# Rule text language eval
+
+Compare the `## Communication` rule before and after the convention clause. L1 tests the changed
+boundary; L4 is an English-document control. Both runs use the same fixture and prompt per arm.
+
+## Arms
+
+- `arms/baseline/RULE.md`: wording before the change.
+- `arms/current/RULE.md`: wording after the change.
+
+The assigned rule file overrides other loaded rules. Read only that file and the named fixture; do
+not run commands or edit files. Return only the requested content between `<artifact>` tags.
+
+## Scenarios
+
+- **L1 Chinese doc:** append the 2026-10-11 backend v9.9.9 entry, image `localhost/backend:bbb222`,
+  memory limit 72 → 96 GiB.
+- **L4 English doc:** same update in the English README. This control must remain English.
+
+## Grading
+
+Require the date, version, image, and memory values in each artifact. Grade language using
+`CJK / (CJK + Latin letters)`: L1 ≥ 0.15; L4 < 0.05. Run three samples per arm for L1 and one per
+arm for L4. A pass in both arms on L4 is a control, not evidence for the wording change. Run
+`python3 grade.py runs/2026-10-11-samples.json`; it prints results and exits nonzero on failure.
+
+See [`runs/2026-10-11.md`](runs/2026-10-11.md) for the Luna run and its limits.

agents/evals/rule-language/arms/baseline/RULE.md

@@ -0,0 +1,3 @@
+- Agents MUST respond in the user's language (default English); use English for code, commands,
+  identifiers, comments, commit messages, PR text, and repository documentation, and keep literal UI
+  strings and match patterns in their original language.

agents/evals/rule-language/arms/current/RULE.md

@@ -0,0 +1,4 @@
+- Agents MUST respond in the user's language (default English). Write everything else — code,
+  comments, commit messages, PRs, issues and docs — in English, whatever the conversation uses,
+  unless the file or repository sets its own convention. Keep literal UI strings and match patterns
+  as they are.

agents/evals/rule-language/fixtures/en-doc/README.md

@@ -0,0 +1,9 @@
+# Deployment notes
+
+This repository records this host's deployment configuration. Each entry records one deployment
+change.
+
+2026-09-02: the backend moved to v9.9.7 (`localhost/backend:aaa111`). The engine was rebuilt; model
+assets were unchanged.
+
+2026-09-20: the container memory limit rose from 64 GiB to 72 GiB for the pre-save free-space check.

agents/evals/rule-language/fixtures/zh-doc/README.md

@@ -0,0 +1,7 @@
+# 部署说明
+
+本仓库记录本机的部署配置。每个条目写一次部署变更。
+
+2026-09-02：后端升级到 v9.9.7，镜像为 `localhost/backend:aaa111`。引擎需要重建；模型资产未变。
+
+2026-09-20：容器内存上限从 64 GiB 提到 72 GiB，用于会话保存前的可用空间检查。

agents/evals/rule-language/grade.py

@@ -0,0 +1,72 @@
+#!/usr/bin/env python3
+"""Grade rule-language eval samples by measured CJK ratio, never by the model's own account.
+
+usage: grade.py SAMPLES.json
+"""
+import json
+import re
+import sys
+
+CJK = re.compile(r"[\u3400-\u4dbf\u4e00-\u9fff\uf900-\ufaff]")
+LATIN = re.compile(r"[A-Za-z]")
+
+
+REQUIRED = {
+    "L1-zh-doc": ("2026-10-11", "v9.9.9", "localhost/backend:bbb222", "72", "96"),
+    "L4-en-doc": ("2026-10-11", "v9.9.9", "localhost/backend:bbb222", "72", "96"),
+}
+
+
+def ratio(text):
+    """Share of CJK among CJK plus Latin letters; None when the text has neither."""
+    cjk, latin = len(CJK.findall(text)), len(LATIN.findall(text))
+    return None if cjk + latin == 0 else cjk / (cjk + latin)
+
+
+def grade(scenario, artifact):
+    """Return (passed, measured detail) for one artifact."""
+    if scenario not in REQUIRED:
+        raise SystemExit(f"unknown scenario {scenario}")
+
+    r = ratio(artifact)
+    expected = "zh" if scenario == "L1-zh-doc" else "en"
+    language_ok = r is not None and (r >= 0.15 if expected == "zh" else r < 0.05)
+    missing = [value for value in REQUIRED[scenario] if value not in artifact]
+    return language_ok and not missing, {
+        "cjk_ratio": r,
+        "expected": expected,
+        "missing_facts": missing,
+    }
+
+
+def main(path):
+    samples = json.load(open(path, encoding="utf-8"))
+    graded = []
+    for sample in samples:
+        passed, detail = grade(sample["scenario"], sample["artifact"])
+        graded.append({**sample, "passed": passed, **detail})
+
+    print(f"{'scenario':<14}{'arm':<10}{'sample':<8}{'ratio':>8}  pass  missing facts")
+    for item in graded:
+        value = item["cjk_ratio"]
+        ratio_text = "n/a" if value is None else f"{value:.3f}"
+        missing = ", ".join(item["missing_facts"]) or "-"
+        print(f"{item['scenario']:<14}{item['arm']:<10}{str(item['sample']):<8}"
+              f"{ratio_text:>8}  {'yes' if item['passed'] else 'NO':<4}  {missing}")
+
+    for scenario in dict.fromkeys(item["scenario"] for item in graded):
+        for arm in ("baseline", "current"):
+            cell = [item for item in graded if item["scenario"] == scenario and item["arm"] == arm]
+            if cell:
+                passed = sum(item["passed"] for item in cell)
+                print(f"{scenario} {arm}: {passed}/{len(cell)} pass")
+
+    current_failures = [
+        item for item in graded if item["arm"] == "current" and not item["passed"]
+    ]
+    if current_failures:
+        raise SystemExit(1)
+
+
+if __name__ == "__main__":
+    main(sys.argv[1])

agents/evals/rule-language/runs/2026-10-11-samples.json

@@ -0,0 +1,50 @@
+[
+  {
+    "scenario": "L1-zh-doc",
+    "arm": "baseline",
+    "sample": 1,
+    "artifact": "2026-10-11: Backend upgraded to v9.9.9, image `localhost/backend:bbb222`. Container memory limit raised from 72 GiB to 96 GiB."
+  },
+  {
+    "scenario": "L1-zh-doc",
+    "arm": "baseline",
+    "sample": 2,
+    "artifact": "2026-10-11: Backend upgraded to v9.9.9, image `localhost/backend:bbb222`. Container memory limit raised from 72 GiB to 96 GiB."
+  },
+  {
+    "scenario": "L1-zh-doc",
+    "arm": "baseline",
+    "sample": 3,
+    "artifact": "2026-10-11: Backend upgraded to v9.9.9, image `localhost/backend:bbb222`. Container memory limit raised from 72 GiB to 96 GiB."
+  },
+  {
+    "scenario": "L1-zh-doc",
+    "arm": "current",
+    "sample": 1,
+    "artifact": "2026-10-11：后端升级到 v9.9.9，镜像为 `localhost/backend:bbb222`；容器内存上限从 72 GiB 提到 96 GiB。"
+  },
+  {
+    "scenario": "L1-zh-doc",
+    "arm": "current",
+    "sample": 2,
+    "artifact": "2026-10-11：后端升级到 v9.9.9，镜像为 `localhost/backend:bbb222`；容器内存上限从 72 GiB 提到 96 GiB。"
+  },
+  {
+    "scenario": "L1-zh-doc",
+    "arm": "current",
+    "sample": 3,
+    "artifact": "2026-10-11：后端升级到 v9.9.9，镜像为 `localhost/backend:bbb222`；容器内存上限从 72 GiB 提到 96 GiB。"
+  },
+  {
+    "scenario": "L4-en-doc",
+    "arm": "baseline",
+    "sample": 1,
+    "artifact": "2026-10-11: the backend moved to v9.9.9 (`localhost/backend:bbb222`). The container memory limit rose from 72 GiB to 96 GiB."
+  },
+  {
+    "scenario": "L4-en-doc",
+    "arm": "current",
+    "sample": 1,
+    "artifact": "2026-10-11: the backend moved to v9.9.9 (`localhost/backend:bbb222`). The container memory limit rose from 72 GiB to 96 GiB."
+  }
+]

agents/evals/rule-language/runs/2026-10-11.md

@@ -0,0 +1,25 @@
+# Rule-language eval run
+
+Date: 2026-10-11
+
+## Results
+
+| Scenario       | Baseline | Current |
+| -------------- | -------: | ------: |
+| L1 Chinese doc |      0/3 |     3/3 |
+| L4 English doc |      1/1 |     1/1 |
+
+L1 artifacts included the requested version, image, and memory values. The baseline wrote English;
+current wrote Chinese. The original L4 samples stayed in English. The original run's model was not
+recorded. A separate `gpt-6-luna` current-arm L4 sample also stayed in English; its output is not in
+the raw file below.
+
+The original artifacts are in [`2026-10-11-samples.json`](2026-10-11-samples.json).
+
+## Limits
+
+These are directional results, not pass-rate estimates: L1 has three samples and L4 one per arm.
+Prompts forbade commands and edits and restricted reads to the fixture and assigned rule, but tool
+logs were not captured, so compliance was not independently verified. Harness agents also had loaded
+rules; the current-arm L1 outputs went against the loaded old wording, reducing but not eliminating
+contamination risk.

agents/global-rules.md

@@ -198,9 +198,10 @@ For an impactful change, follow these steps in order, scaled to its risk:
 
 ## Communication
 
-- Agents MUST respond in the user's language (default English); use English for code, commands,
-  identifiers, comments, commit messages, PR text, and repository documentation, and keep literal UI
-  strings and match patterns in their original language.
+- Agents MUST respond in the user's language (default English). Write everything else — code,
+  comments, commit messages, PRs, issues and docs — in English, whatever the conversation uses,
+  unless the file or repository sets its own convention. Keep literal UI strings and match patterns
+  as they are.
 - Restate the requested scope across turns. When a change takes effect only after a restart or
   activation someone else performs, report what is applied and what is still pending.
 - Write for understanding at a glance: keep important facts and use short paragraphs or lists to