Subscribe
0%
Illustration for: METR's new yardstick: AI task horizons now double every ~3.6 months, and they're getting harder to measure

METR's new yardstick: AI task horizons now double every ~3.6 months, and they're getting harder to measure

METR's November 20 capability update swaps its saturated task suite for longer, messier tasks. It finds frontier agents' 50% time horizons doubling roughly every 3.6 months, with up to 27-hour horizons on verifiable work. It also warns that easy-to-run evals may now overstate real-world skill.

Listen to this article0:00 / 5:36

METR, the nonprofit behind AI's most-quoted "time horizon" chart, published a major update on November 20. On a combined set of old and new tasks, it finds frontier agents' 50% time horizon doubling roughly every 3.6 months. That's the length of task, measured in skilled-human hours, that a model completes half the time. METR's own headline is just as notable: AI R&D-relevant capabilities are improving fast and getting harder to measure.

Why it matters: labs, investors and policymakers use METR's chart to judge how close AI is to automating real research work. The update says the old ruler has run out, and the new one is costlier, noisier and leans more on human judgment.

What changed

METR says its Time Horizon 1.1 suite, released in March 2025, was saturated within about a year. At launch the best model managed about an hour; the suite's tasks top out around 30 hours. The update measures roughly 15 models, from GPT-4o through Opus 5.5 and GPT-6 Astra, on three suites:

Suite What it is
Time Horizon 1.1 222 tasks drawn from SWAA, HCAST and RE-Bench
MirrorCode 75 command-line software-replication tasks
TH2 (new) 36 tasks in 12 families: 8 "verifiable" (optimisation, ML, puzzles, games) and 4 "hard-to-verify & judgment"

TH2 tasks are scored at several thresholds, each paired with an estimated human time, and the longest threshold is 200 hours. Agents ran in a ReAct-style scaffold at high reasoning effort. METR also introduces a Lifecycle Cost metric (what it costs to build a task, then run and score it 100 times) and a composite METR Capability Index.

The numbers

  • Doubling time: about 3.6 months across TH1.1 and TH2 combined. Pebblous, quoting the report, gives the precise figure as 3.61 months with a 95% interval of 2.08–4.94 months.
  • Verifiable tasks only: METR says doubling is faster on this subset. Pebblous puts it at 1.65 months (95% CI 0.79–3.99). That is a narrower slice, not the headline trend.
  • Current ceiling: frontier models reach horizons of up to 27 hours on the TH2 verifiable subset, METR says.
  • Extrapolation: a naive extrapolation puts the 50% horizon at the 200-hour threshold in July 2027. Given the wide interval, treat that date loosely.
  • Cross-check: according to Pebblous, the report cites a 3.8-month doubling on OpenAI's internal research tasks, which is similar to METR's own result.
  • MirrorCode: Opus 5.5 solves 77% of medium-to-large programs, per METR.
  • Against humans: the strongest models (Fable 5.1, Opus 5.5, GPT-6 Astra) placed in the 96th–98th percentile of human entrants across 15 AZsPCs optimisation contests. In the NanoGPT speedrun, the best agent run got a 15.7% speedup in 48 hours, against 41% for the top human record after about 60 hours of AI-assisted work.

Why it's getting harder to measure

METR lists several problems with its own method:

  • Few real baselines. Only about 9% of the human-time estimates come from people actually doing the tasks; researchers estimated the rest. Baseliners were paid $200–350 an hour, and seven attempts cost about $90,000.
  • Cost skews results. On tasks with higher Lifecycle Cost, models look weaker. METR estimates that each doubling of Lifecycle Cost pushes back the 50% success point by about 4.6 months, holding task length fixed. Cheap, containerised tasks may flatter models.
  • Overestimation risk. METR says it is "increasingly concerned" its easy-to-run evals overestimate real-world capability, partly because of overlap with the kinds of tasks models are trained on.
  • Judgment gap. On tasks without clear feedback, agents showed worse judgment than experts, Remio notes. Model graders also overrate strong models: per Pebblous, Opus 5.5 scored GPT-6 Astra's outputs about 0.2 points (on a 0–1 scale) higher than METR's human rater. When METR asked agents to build new eval tasks, none were good enough to use.

METR's composite index shows no clear acceleration or slowdown across the public frontier, according to Remio's reading of the report. METR adds that public evidence can't rule out drastic AI R&D acceleration in the next couple of years.

Read it carefully

Per-model horizons appear in the report as charts. We couldn't verify individual figures, so we haven't printed any. Some coverage, including Korean headlines Pebblous flags, has turned the update into a single "hours per model" number. METR's own wording is more careful: the trend is steep, the error bars are wide, and the tasks now cost far more to build and grade.

What to watch

The 200-hour threshold is the next marker. If METR's next update shows models clearing large parts of TH2's judgment-heavy families, not just the verifiable ones, the July 2027 extrapolation will start to look less naive.

Sources

  1. METR — AI R&D-relevant model capabilities are improving rapidly and are getting harder to measure metr.substack.com
  2. Pebblous — METR Q4 2026 Capability Update: Time Horizon and Lifecycle Cost blog.pebblous.ai
  3. Remio — METR Update Reveals a Gap Between AI Coding and Judgment remio.ai
  4. roomedia/ax-trend — METR Q4 2026 capability update digest (GitHub issue #4100) github.com

#METR#AI evaluation#time horizon#AI agents#Opus 5.5#GPT-6 Astra#AI R&D