HarnessPro tunes the harness around a coding agent, not the model weights. It runs the agent many times, reads the failures, and rewrites instructions, tools, hooks and context handling. Every candidate is scored on pass rate and dollars per task, so the output is a few harnesses at different budgets instead of one winner. On GPT-5.5 with Codex, the accuracy-first version beat RHO by 2.0 points on average while costing 26.1% less per task. The balanced version on SWE-Bench Pro gave up one point and cut cost 53.0%, from $2.68 to $1.26.
The layer that decides what the model can do
A harness is the runtime around a model: the system prompt, tool definitions, loadable skills, context assembly, memory, runtime hooks, and the rules for calling a run finished. Swap the harness and the same model behaves like a different product.
Self-improvement methods already rewrite this layer from past runs. AFlow evolves workflows, SkillMOO evolves skills, and RHO, the closest baseline here, rewrites skills and tools. All three buy accuracy with more attempts, more exploration, more checking. The bill grows with the score.
HarnessPro keeps the loop and changes what counts as a win. A candidate is scored on pass rate and on cost per task, and one that loses on both is discarded. The survivors sit on a Pareto front: Lite for tight budgets, Medium in the middle, Heavy for accuracy.
How one round of evolution works
An epoch starts with a set of candidate harnesses. Each one runs several times on the same training tasks, recording pass or fail, tokens, latency and tool calls. Then three agents take over.
- A trajectory analyzer reads several rollouts of one task side by side. It pinpoints the failure, the wasted exploration or the repeated work, and keeps the evidence for each finding.
- A planner writes three plans: cost first, balanced, accuracy first. An editor implements each in its own workspace and commits the version to Git.
- Candidates run against their parents. Anything below the starting pass rate, or dominated on both metrics, is dropped. Survivors become parents for the next epoch.
What gets edited is prompts, tool config, hooks, context rules. Weights never move. That is why the result can sit on top of whatever model you already pay for.
Why a generated test suite replaces the judge
You cannot use official tests during evolution. The search would overfit them. RHO asks an LLM to compare trajectories instead, and the HarnessPro authors found those judges too generous: they pass implementations that are broken.
HarnessPro writes its own tests. A synthesizer reads the task description and the first trajectory, produces a fixed suite, and every later candidate faces exactly that suite. Failure recall moves from 41.7% with an LLM judge to 83.3% with generated tests. The paper is careful about the limit. Synthesized tests are not always right. They are just better at surfacing a failure than asking a model for a verdict.
The measured numbers
Terminal-Bench 2 covers terminal work, SWE-Bench Pro covers repository-level fixes, ProgramBench covers rebuilding programs. The main run pairs GPT-5.5 with Codex, and cost is computed from token usage at list prices.
| Setting | Terminal-Bench 2 (pass / cost) | SWE-Bench Pro (pass / cost) | ProgramBench (pass / cost) |
|---|---|---|---|
| Vanilla | 69.5% / $0.89 | 59.0% / $1.88 | 60.3% / $2.65 |
| RHO | 76.3% / $1.09 | 80.0% / $2.68 | 68.8% / $3.59 |
| HarnessPro Lite | 72.9% / $0.43 | 72.3% / $0.80 | 64.3% / $2.18 |
| HarnessPro Medium | 76.3% / $0.56 | 79.0% / $1.26 | 67.9% / $2.90 |
| HarnessPro Heavy | 80.8% / $0.72 | 80.7% / $1.97 | 69.6% / $2.95 |
GPT-5.5 with Codex. Pass rate is the mean of three runs; cost is mean dollars per task.
Heavy has the highest pass rate on all three benchmarks and still costs less than RHO. On Terminal-Bench 2 it hits 80.8% against 76.3%, at $0.72 per task instead of $1.09. On SWE-Bench Pro, Medium gives up one point, 79.0% against 80.0%, and cuts cost by 53.0%. ProgramBench is nearly a tie on accuracy, 69.6% against 68.8%, with $2.95 against $3.59.
The same shape shows up with GLM-5.2 and Claude Code. Heavy reaches 81.4% on Terminal-Bench 2 at $0.62 per task, where RHO gets 78.0% at $1.00. Lite beats the plain agent on all three benchmarks while spending less, which is the strongest evidence that the gain is not tied to one model stack.
What the evolved harnesses actually changed
The authors list the behaviors the search settled on. None of them are tricks.
- Pick a reasoning level and a tool budget from the task description instead of using one setting for everything.
- Add runtime hooks that tell the model to stop re-checking work it already validated.
- Compress long tool output and drop repeated instructions while keeping the checks that matter.
Cost falls because waste falls. The agent still does the work that matters, which separates this from the cheap shortcut of forcing every task into low reasoning effort. The ablations make that difference visible.
What happens when you remove the parts
Take out the test synthesizer and pass rate drops 3.4 to 10.0 points across both benchmarks and all three modes. Take out the planner and the damage shows on both axes. On SWE-Bench Pro, Medium falls from 79.0% to 73.0% while cost climbs from $1.26 to $1.42.
Copying one trick into RHO does not reproduce it. Dynamic routing alone or budget hooks alone give a partial gain. Pinning every task to low reasoning effort saves money and hurts accuracy. The loop is doing the work, not any single component.
What this means if you are deploying agents
Agent tuning usually optimizes accuracy and finds out about cost at the end of the month. HarnessPro treats both as design inputs and hands back a menu. For a team running agents at volume, the menu is more useful than a single best score, because the right operating point depends on the task mix and the budget.
Two caveats. Three benchmarks and two model stacks are a narrow base, so expect the size of the gain to shift on your own traffic. And generated tests can be wrong. Treating them as ground truth rebuilds the problem this method exists to avoid.
The loop does not need the authors' code. It needs a harness you can version, a way to replay tasks, and a score that counts tokens alongside outcomes.
Harness gains compound with whatever model sits behind them, and comparing models is easier when the keys and billing live in one place. The SaveMyToken multi-model plan puts several agent-capable models behind one endpoint, so you can move a workflow between them without opening a new account for every provider.
Primary source: Zhuoshi Pan et al., HarnessPro: Recursive Harness Self-Improvement with Pareto Optimization for Performance-Cost Trade-offs, arXiv:2610.26770, October 2026. Benchmark figures are as reported in the paper; SaveMyToken has not rerun them.
Quick answers
Is HarnessPro a new model?
No. Model weights stay frozen. The method edits the runtime layer around the model: instructions, tools, context assembly, memory, hooks and verification rules.
Do you need the authors' harness to use the idea?
No. It applies to any harness you can version and edit. The reusable part is the two-objective scoring and the three-stage loop, not the specific codebase.
Does a lower per-task cost mean lower quality?
Not in these results. Lite, Medium and Heavy sit on the same Pareto front, and each one undercuts RHO on cost while matching or beating its pass rate on at least one benchmark. Choose the mode that clears your accuracy floor.