TL;DR

Arena's HarnessTax study compares 21 model-and-agent combinations across sampled software and terminal tasks. Its useful distinction is between the model and the software assembling context, tools and execution around it: similar success can carry different costs. Read the task sample, attempts, settings and pricing snapshot with the results. The study does not establish a universal best or cheapest agent.

What to look for

Arena compares 21 combinations across Claude Code, Codex CLI, and Pi. The study shows why model selection also involves the software managing its tools and context: similar success rates can come with different costs.

Keep in mind

The experiment samples 30 tasks each from SWE-bench Lite and Terminal-Bench 2.0, with three attempts per combination and task. Its pricing snapshot and harness settings matter. These results do not establish a universal winner for other workloads.

Read the original work

HarnessTax: How Much Does the Harness Matter for Coding Agents? ↗

By Arena Team · Arena

Published Sep 16, 2026. Source checked Oct 5, 2026.

Reading note by West Mao and Alex George · Added Oct 5, 2026. This page is a short editorial introduction; the full article belongs to its original publisher.