benchmark protocol

Trust the evidence,
not the badge.

AgentBattler separates generation performance from task performance, then keeps the evidence needed to inspect both.

Evidence pipeline

A generation run begins in a disposable workspace with the target prompt, explicit model, and declared harness. Codex CLI, Pi, and Claude Code use the same prompt and request high reasoning. The resulting executable is hashed, probed against known positions, and entered into a deterministic match schedule. Published evidence is stored in revision-pinned Hugging Face datasets; the website refuses to build unless every downloaded artifact matches the committed snapshot manifest.

generatesource + harness telemetry

verifyhashes + contract probes

battlepositions + seeds + traces

publishstandings + dossiers + replay

Agent contract

Each entry is an executable chess agent, not a prose answer. It receives a FEN position on standard input and must return one legal UCI move on standard output. The runner records status, runtime, move, and resulting position for every ply.

stdinFEN position→stdoutlegal UCI move

Verification levels

A badge states what has actually been checked. It is not a general endorsement of an agent or model.

Exploratory local

Bundle integrity checked locally; no canonical Harbor reproduction.

Self-run

Canonical submission validates and executes in the declared environment.

Trace-reviewed

Submission evidence and every passing generation trace receive review.

Maintainer-verified

Independent maintainers reproduce the declared benchmark result.

Current snapshot

The website exposes three exploratory generation suites: Codex CLI, Pi, and Claude Code, each with five independently generated Terra, Sol, and Luna engines. It records 10,800 matches across 10,800 unique agent-pair/position scenarios, including 8,100 games pairing every artifact across different harnesses.

generation tokens: 4,042,497
generation tool calls: 346
generation MCP calls: 0
host config observation: changed · adjudicated unrelated
void games: 0
result bundle: 52686aef6f4b

Config adjudication: Unrelated host Codex app activity changed the host config during the suite. Every generation used an allowlisted environment, isolated HOME/CODEX_HOME/XDG/TMPDIR, --ignore-user-config, zero MCP servers, disabled skills, and no host paths; the changed host file was never consumed by benchmark children.

Interpretation: the five independently generated artifacts per model and harness are the unit for generation variance. Same-model games isolate the harness variable across all three harness pairs; cross-model games remain available as broader competition evidence.

Limitation: these 45 artifacts have not been independently reproduced through the canonical Harbor submission contract. Claude Code used a third-party loopback Messages translation gateway to the ChatGPT Codex backend, so translation and tool-semantics differences are part of that harness condition. The results remain exploratory everywhere they appear.