Fancy harnesses make coding agents more expensive but not better, study finds
What changed
A controlled study tested various coding-agent harnesses like Claude Code, OpenCode, and mini-SWE-agent to see if switching among them improves coding pass rates. The results showed very little difference in success rates between changing harnesses and simply rerunning the same setup multiple times. However, the cost per task increased significantly, sometimes tripling. This cost inflation is mainly due to repeatedly resending the system prompt at every coding step.
Why builders should care
Many development teams rely on AI agents for coding assistance and often assume that more sophisticated or specialized harness frameworks will deliver better results. This study exposes that those fancy harnesses do not improve output quality or pass rates in a meaningful way. Instead, they drive up operational costs without corresponding benefits. For builders managing tight budgets or scaling AI-assisted workflows, understanding this trade-off is crucial to avoid unnecessary spending.
The practical takeaway
Switching between different coding-agent frameworks for the sake of diversity or sophistication does not yield better coding performance. Rerunning the same setup multiple times is almost as effective. The biggest culprit in cost spikes is the system prompt, which is resent at every step. Builders should focus their optimization efforts on reducing prompt size or frequency rather than chasing different harnesses. This can lead to better cost efficiency without sacrificing result quality.
What to watch next
Keep an eye on improvements in prompt engineering and agent orchestration that cut down redundant information exchange. Tools that optimize system prompts or enable stateful context sharing rather than full prompt resends could reduce costs dramatically. Also, developments in lightweight harnesses or agent caching methods might address current inefficiencies. The business case for complex agent frameworks is under pressure until they prove real gains beyond cost.
AI Quick Briefs Editorial Desk