# Same model, up to 5x the bill: FrontierHarness v1.1 tests 22 coding-agent harnesses across 27,280 runs

> Runta's new benchmark ran five open models through 22 coding-agent harness setups and found the wrapper can multiply cost per task by up to 5x with little or no gain in pass rate, and that the cheapest harness for one model can be among the priciest for another. Here's what the numbers show and where they stop.

Published Nov 18, 2026 by TechMeasured Staff · AI · Canonical: https://techmeasured.com/ai/frontierharness-v1-1-coding-agent-harness-costs-5x-same-model-27280-runs-what-it-shows

Runta, which sells agent execution infrastructure, published **FrontierHarness Eval v1.1** on November 17, a benchmark of coding-agent "harnesses" — the software (Codex CLI, OpenCode, Cline CLI and others) that wraps a model, feeds it tool calls and decides when it's done. The headline finding: with the model held fixed, the most expensive harness cost **1.8 to 5.0 times** the cheapest per task, usually without passing more tasks.

Why it matters: teams pick agent wrappers mostly by habit or features, then blame the model when bills climb. This is one of the largest public attempts to separate the two, and it suggests the "best" harness isn't a fixed answer — it depends on which model you plug in.

## How the test worked

According to AI/TLDR's summary and Felipe Castillo's analysis on DEV Community, both drawing on Runta's report:

- **5 open models**: GLM-5.3, GLM-5.3 Flash, DeepSeek V4.1 Flash, Muse and Kimi-K3, all served by Baseten at list prices.
- **22 harness configurations** (18 harnesses; four DeepSeek Harness profiles and two OpenCode versions counted separately), run through the Harbor framework at default settings and maximum reasoning effort.
- **62 terminal tasks × 4 attempts**, for 27,280 runs and about 83 billion tokens, each in a fresh, isolated runtime. The project's live page says tasks span software engineering, science, data, systems, ML and security.
- Only verifier passes count. **216 trials** flagged for reward hacking were scored zero — 163 of them for looking answers up online, per Castillo.

That is roughly 76 times the size of v1.0, which tested nine harnesses on Kimi-K3 alone, AI/TLDR says.

## What the numbers show

| Finding (Runta data, as reported) | Figure |
|---|---|
| Share of run-cost variance explained by the model | 92.8% |
| …by the harness alone / by the model-harness pairing | 2.3% / 4.9% |
| Most vs least expensive harness per task, same model | 1.8x–5.0x |
| Spend on runs that didn't pass | 64.5% (52.7% failures, 11.8% timeouts) |
| Median harness's cost-rank shift when the model changes | 11 of 22 places |
| Best pass rate (Pi, Codex CLI, ZCode, tied) | 61.9% |
| Lowest cost per task (OpenCode v2) | $0.561 |

**Rankings don't transfer.** Codex CLI had the lowest cost per pass with GLM-5.3 ($0.803, per AI/TLDR) but ranked 16th with Kimi-K3. DeepSeek Harness's Creator profile was 1st with Muse and 21st with GLM-5.3 Flash; OpenCode v1 was 1st with DeepSeek and 13th with Muse. Castillo reports a split-half rank correlation of 0.71 within a model but just 0.26 across models.

**Paying more rarely bought more passes.** The widest gap was on DeepSeek, where Cline CLI cost 5.0x OpenCode v1 per task and passed 2.0 points more often. On the other four models the cheapest harness trailed the priciest by at most 0.4 points. The harnesses tied for the best pass rate ranked anywhere from 5th to 20th on cost per task.

**Steps, not prompts, drive cost.** The number of agent steps correlated +0.51 with cost per pass; starting-prompt size, tool count and cache-hit rate were indistinguishable from noise, Castillo writes. mini-SWE-agent had the smallest starting prompt (882 tokens) but averaged 38 steps per pass and ranked 17th of 22.

**Agents fail confidently.** Among the five best-value harnesses, 68–82% of failed runs ended with a final answer rather than a timeout — the agent believed it was done. One example: Codex CLI on GLM-5.3 reported successful validation on an async-cancellation task the verifier then failed.

## Caveats

- **Vendor benchmark.** Runta sells the execution layer the trials ran on; Baseten served the models and Z.ai is listed as a sponsor on the project site.
- **Open models only.** No Claude or GPT models were tested, and Runta notes closed-model results may differ.
- **One provider, list prices, defaults.** Different caching, pricing or tuned settings could reorder the table.
- **Statistical noise.** With 62 tasks, each pass rate carries a 95% interval of about ±10 points, so small gaps are ties.
- We could not load Runta's own blog post; figures here come from the two independent write-ups and the project site.

## What to do

If you run coding agents at volume on open models, benchmark your harness *with your model* before standardising — a leaderboard result for another model tells you little. Track steps per task and the share of spend going to failed runs; those predicted cost here far better than prompt size. And treat an agent's "all tests pass" as a claim to verify, not a result.


## Sources

1. [AI/TLDR — FrontierHarness Eval v1.1: which agent harness…](https://ai-tldr.dev/releases/runta-frontierharness-eval-v1-1/)
2. [DEV Community — Why Your Coding Agent Costs 5x More Than It Should (It Is Not the Model)](https://dev.to/castillodk/why-your-coding-agent-costs-5x-more-than-it-should-it-is-not-the-model-31lm)
3. [FrontierHarness Eval — live benchmark run](https://frontierharness.org/live/)
