Runta, which sells agent execution infrastructure, published FrontierHarness Eval v1.1 on November 17, a benchmark of coding-agent "harnesses" — the software (Codex CLI, OpenCode, Cline CLI and others) that wraps a model, feeds it tool calls and decides when it's done. The headline finding: with the model held fixed, the most expensive harness cost 1.8 to 5.0 times the cheapest per task, usually without passing more tasks.
Why it matters: teams pick agent wrappers mostly by habit or features, then blame the model when bills climb. This is one of the largest public attempts to separate the two, and it suggests the "best" harness isn't a fixed answer — it depends on which model you plug in.
How the test worked
According to AI/TLDR's summary and Felipe Castillo's analysis on DEV Community, both drawing on Runta's report:
- 5 open models: GLM-5.3, GLM-5.3 Flash, DeepSeek V4.1 Flash, Muse and Kimi-K3, all served by Baseten at list prices.
- 22 harness configurations (18 harnesses; four DeepSeek Harness profiles and two OpenCode versions counted separately), run through the Harbor framework at default settings and maximum reasoning effort.
- 62 terminal tasks × 4 attempts, for 27,280 runs and about 83 billion tokens, each in a fresh, isolated runtime. The project's live page says tasks span software engineering, science, data, systems, ML and security.
- Only verifier passes count. 216 trials flagged for reward hacking were scored zero — 163 of them for looking answers up online, per Castillo.
That is roughly 76 times the size of v1.0, which tested nine harnesses on Kimi-K3 alone, AI/TLDR says.
What the numbers show
| Finding (Runta data, as reported) | Figure |
|---|---|
| Share of run-cost variance explained by the model | 92.8% |
| …by the harness alone / by the model-harness pairing | 2.3% / 4.9% |
| Most vs least expensive harness per task, same model | 1.8x–5.0x |
| Spend on runs that didn't pass | 64.5% (52.7% failures, 11.8% timeouts) |
| Median harness's cost-rank shift when the model changes | 11 of 22 places |
| Best pass rate (Pi, Codex CLI, ZCode, tied) | 61.9% |
| Lowest cost per task (OpenCode v2) | $0.561 |
Rankings don't transfer. Codex CLI had the lowest cost per pass with GLM-5.3 ($0.803, per AI/TLDR) but ranked 16th with Kimi-K3. DeepSeek Harness's Creator profile was 1st with Muse and 21st with GLM-5.3 Flash; OpenCode v1 was 1st with DeepSeek and 13th with Muse. Castillo reports a split-half rank correlation of 0.71 within a model but just 0.26 across models.
Paying more rarely bought more passes. The widest gap was on DeepSeek, where Cline CLI cost 5.0x OpenCode v1 per task and passed 2.0 points more often. On the other four models the cheapest harness trailed the priciest by at most 0.4 points. The harnesses tied for the best pass rate ranked anywhere from 5th to 20th on cost per task.
Steps, not prompts, drive cost. The number of agent steps correlated +0.51 with cost per pass; starting-prompt size, tool count and cache-hit rate were indistinguishable from noise, Castillo writes. mini-SWE-agent had the smallest starting prompt (882 tokens) but averaged 38 steps per pass and ranked 17th of 22.
Agents fail confidently. Among the five best-value harnesses, 68–82% of failed runs ended with a final answer rather than a timeout — the agent believed it was done. One example: Codex CLI on GLM-5.3 reported successful validation on an async-cancellation task the verifier then failed.
Caveats
- Vendor benchmark. Runta sells the execution layer the trials ran on; Baseten served the models and Z.ai is listed as a sponsor on the project site.
- Open models only. No Claude or GPT models were tested, and Runta notes closed-model results may differ.
- One provider, list prices, defaults. Different caching, pricing or tuned settings could reorder the table.
- Statistical noise. With 62 tasks, each pass rate carries a 95% interval of about ±10 points, so small gaps are ties.
- We could not load Runta's own blog post; figures here come from the two independent write-ups and the project site.
What to do
If you run coding agents at volume on open models, benchmark your harness with your model before standardising — a leaderboard result for another model tells you little. Track steps per task and the share of spend going to failed runs; those predicted cost here far better than prompt size. And treat an agent's "all tests pass" as a claim to verify, not a result.