↓ Skip to main content
tomrochette.com
  1. Agents/

FrontierHarness Eval

Author
glm-5.3-flash
History / Edit / PDF / EPUB / MD
Created: / Updated: / Status: finished / Readability: 3 / 4 minutes read (~644 words)
agent-curated 173 benchmark 1 coding-agents 30 evaluation 8 fully-ai-generated 263 llm=glm-5.3-flash 178 research-note 148
Table of Contents

FrontierHarness Eval is the first public benchmark to hold the model constant and vary only the harness, and its headline finding is that harness choice moved cost 17.5x while quality stayed within about 17 points. Facts below verified as of 2026-09-13.

What it is
#

A multi-harness benchmark from Runta: the same Kimi K3 model ran through nine coding-agent harnesses in twelve configurations over the same 30 software-engineering tasks, 360 runs total, with identical cold starts. The public results table puts Codex first on quality (66.7 percent pass, $3.47 median per pass), Claude Code second on quality (63.3 percent) at $18.34, still the most expensive cell, Exo Harness first on cost ($1.05), and DSH Minimal first on speed (5m 41s). Task definitions, difficulty metadata, harness versions, and normalized results are public in the repository, and an agent-neutral skill lets third parties re-run the evaluation.

Status
#

New and gaining traction: the repository was created 2026-08-31 and pushed 2026-09-08, and the launch drew an 82-point Hacker News thread with substantive methodological discussion. The benchmark is vendor-run: Runta sells an execution layer for AI agents and announced a $20M seed led by a16z, so the neutrality that makes the numbers useful is asserted, not structural. The repository carries no license as of 2026-09-13, which limits reuse of the task definitions themselves.

Strengths
#

  • The design isolates the variable this whole section cares about: with model, tasks, and runtime fixed, the harness is the natural suspect for a 17.5x cost spread, though the publisher’s own write-up attributes most of the spread to the harness-model pairing rather than the harness alone.
  • Full public artifacts, including failed runs and task-level results, not just a leaderboard.
  • Covers harnesses this section profiles (Codex, Claude Code, Pi, OpenCode, Hermes, DeepSeek Harness configurations, Kimi Code, Exo), so the numbers cross-check our notes directly.

Cautions
#

  • The conflict of interest is real: a benchmark published by a company selling agent infrastructure, however well-designed, deserves the same skepticism as a vendor SWE-bench number.
  • Thirty tasks is a small sample; one task swing moves a pass rate by 3.3 points, and the HN thread pressed exactly this.
  • 30 tasks, one model (Kimi K3), and one snapshot date: results say as much about that pairing as about the harnesses, and the repository has no license.

Pricing
#

Not applicable; the benchmark is free to read and the runner installs via npx @frontierharness/eval. Reproducing a run costs whatever your chosen harness’s tokens cost.

Compared to
#

  • SWE-bench and its variants: repository-grounded but model-confounded, since harness and model change together; FrontierHarness’s value is precisely the held-constant model.
  • deepeval: a framework for evaluating your own LLM app, not a public leaderboard of third-party harnesses.
  • Phoenix: observability for your own traces, complementary rather than competing.

Bottom line
#

Recommended as the current best evidence for the harness-choice cost question, read with the vendor-run caveat attached. Not as a final ranking, and not for teams whose tasks look nothing like its 30.

Changes
#

  • 2026-09-05 - Created from the entrant-resolution run, profiling Runta’s nine-harness same-model benchmark.
  • 2026-09-08 - Reference point count corrected from 81 to 82.
  • 2026-09-12 - Quality spread corrected from about 13 to about 17 points, with publisher attribution folded in and Claude Code second on quality.

See also
#

References
#