Two and a half years after the Devin demo went viral at 13% on SWE-bench, Cognition president Russell Kaplan sits down with Harrison Chase for the most candid accounting yet of what running coding agents at enterprise scale actually costs — and why the industry’s incentives are quietly pointed in the wrong direction. The central claim: a lot of the ecosystem is structurally motivated to get customers to token-max, and the bill is now coming due.
From viral demo to migrations
The origin story is more mundane than the demo suggested. Devin launched March 2024 at “the edge of possible.” June 2024, Devin became the number-one committer to Devin — the first real internal milestone, reached through months of manual dogfooding. The early external product-market fit wasn’t general software engineering at all; it was migrations and refactors at large enterprises.
The reason is economic rather than technical. For a 10,000-module refactor it’s worth investing real effort in getting the prompt, setup, and context exactly right, because you amortise that effort across every module. You get high ROI without needing general software intelligence.
“There is no best model” — capability saturation
The most quotable structural argument in the conversation: the step changes in model capability have gotten so large that capability is no longer the axis developers care about.
“More and more of the tasks in software engineering are getting intelligence-saturated.”
Kaplan’s framing is that for adversarial workloads — anything where you need to out-think a counterparty — you always want maximum frontier intelligence. But for most software people actually build, there’s a saturation threshold: once it’s good enough, what you care about is speed and cost. Developer sensitivity has shifted from “I need the best model” to “I’m spending so much money on my coding agents.”
Where do frontier models still matter? Kaplan is specific:
- Security vulnerability detection and remediation — the Fable-class models are strong on precision, GPT-5.5/5.5-Cyber are better on recall. The best harness uses both and filters one through the other.
- Processing messy integration data (Datadog and similar) — volume plus multi-step reasoning, and visibly the product of heavy RL on realistic data sources.
Frontier Code: measuring mergeability, not correctness
SWE-bench is saturated. When Cognition went looking for an eval that matched their internal intuition that a model “feels a lot better,” they found the gap was mergeability:
“This code is technically correct, but would you actually merge it? Would this improve the quality of your codebase?”
The construction process is worth studying for anyone building domain evals:
- A recruiting campaign with leading open-source maintainers to encode their standards for what makes a PR acceptable.
- Scoring split into blocking (binary — tests pass, hard deterministic maintainer criteria) and non-blocking (a linearly weighted aggregate of stylistic criteria, including LLM-as-judge heuristics).
- A recurring code smell they penalise: agents making the right change but also touching files they shouldn’t. Correctness score unaffected; style score down-weighted.
- Reverse evaluation: not just “tests pass after the change” but “tests fail without the change” and “tests fail under this other change.”
- Every single researcher on the team hand-contributed and reviewed eval items. “This is not something you can throw over the wall.”
Each item is hundreds of hours of work — “a full project and labour of love per question.” The Frontier Code Diamond subset sat in the teens pre-Fable 5 and is now in the 30s.
There’s a striking aside on where this ends up. Kaplan started in ML on Tesla’s autopilot team in 2017, when the bottleneck was obviously GPUs. Today, he says, the bottleneck at Tesla is running the evals — interventions are so rare you need enormous driving volume to find any problem at all. Software engineering isn’t there yet, “but we might hit that threshold sooner than we think.”
Devin Fusion: routing and sidekicks
The cost problem is now acute enough to name plainly:
“There are organizations where the per-person token spend is starting to eclipse the human salary spend.”
Kaplan’s analogy: it would be considered insane if anyone at your company could hire a thousand people tomorrow without asking. That is roughly how we run agent fleets today.
Devin Fusion does both routing and parallel execution. The core mechanism is a sidekick agent: a frontier model executes the task while a cheaper, more price-performant model runs the same task in parallel, with the frontier model deciding when to delegate. Running both in parallel keeps context alive on both sides, with frequent filesystem writes when context overflows. Result: ~35% better price-performance with a slight increase in quality.
An observation that generalises well beyond Cognition: delegation is an emergent capability. As frontier models get smarter, one of the key skills that improves is knowing which subtasks can safely go to a dumber model — mirroring human career progression.
On their own models: SWE-1.6 is the most-consumed model by tokens in Devin Desktop, around Opus 4.6 level. Kaplan’s argument for startups training their own is that specialisation buys you frontier capability at a given moment for a specific workload, and you accept the model has a 3–6 month half-life. Cognition were also the first Western firm to deploy Cerebras at scale, hitting ~950 tokens/sec on their own models.
Proactive agents and the CTO-of-10,000-agents shift
The biggest internal change Kaplan reports isn’t a model — it’s automations: wiring agents to be proactive rather than reactive. Devin triages every message in a set of Slack channels, deciding whether to chime in or investigate. Bug reports, crashes, user feedback all get handled by default; humans move up to deciding which bets change the company’s trajectory.
He explicitly rejects “software factory” as the term for this — a factory produces the same thing repeatedly, whereas the value of coding agents is that the output is bespoke and fully contextual.
The most acute expression is Devin Security Swarm, described as agentic map-reduce for finding and fixing vulnerabilities. Because every Devin session runs in its own microVM, you can safely reproduce a potential vulnerability, replicate it, and validate the fix. A codebase too large for any single context window gets sharded during the scanning phase, then fixes are aggregated back. One person fires an enterprise-wide API call doing the equivalent of thousands of engineers’ work.
For new grads worried about the ladder: Kaplan argues the relative advantage actually favours them, because everyone has barely any experience working with agents, and onboarding has never been easier — you can ask an agent every silly question without judgement.
The $10M productivity guarantee
This is the part that makes the episode’s title land. The stated problem:
“A lot of people in the industry have a strong incentive to get customers to token-max… and at some point the bill comes due.”
Enterprises that spent a year token-maxing are now at tens or hundreds of millions annually, with leadership asking what they got. Cognition’s position as an independent agent lab, Kaplan argues, structurally aligns them with the customer rather than with driving usage of any one model.
Their approach to quantifying value:
- Concede that automatic ROI estimation is impossible in 2026 — you don’t have the business context to know what a feature is worth.
- Measure one level below: productive engineering output. Merged PR → productive. PR created but never merged → conservatively counted unproductive. No PR at all (data analysis and similar) → ML classification against a manually tagged, customer-collaborative dataset.
- Separately survey customers to estimate engineering hours saved per equivalent session.
Combined, that yields an evaluator agent scoring every session for productivity and hours saved — enough confidence to financially underwrite a $10 million productivity guarantee: if you pay more than the dollar value of engineering hours you got, Cognition refunds the difference.
The internal pushback was obvious — what if someone runs Devin in a loop and burns millions? The answer was to add guardrails and admin budget controls, which is the honest version of incentive alignment: make the failure mode structurally impossible rather than promise it won’t happen.
The most-used feature of the resulting dashboards turned out not to be cost policing but coaching — teams comparing efficiency, and Devin telling you directly that your prompt was under-specified.
Key takeaways
- Capability saturation has flipped the buying question from “which is the best model” to “what’s the cheapest model above threshold” for most software work.
- No single model wins. Fable-class leads on security precision, GPT-5.5 series on recall — the best harness runs both and filters.
- Mergeability is the eval frontier. Correctness is saturated; whether a human maintainer would accept the PR is not.
- Split eval scoring into blocking and non-blocking criteria, and add reverse evaluation — tests should fail without the change.
- Evals can’t be outsourced. Every Cognition researcher hand-authored and reviewed items; each is hundreds of hours of work.
- Sidekick parallelism beat pure routing — ~35% better price-performance with a slight quality gain.
- Delegation is an emergent model capability that improves with frontier intelligence, mirroring human seniority.
- Per-person token spend is starting to eclipse salary spend in some organisations — and a $10M productivity guarantee is a bet that measured hours-saved can be underwritten.
Source
- Talk: The misaligned incentives behind AI coding agents (Max Agency podcast)
- Speakers: Russell Kaplan (President, Cognition) with Harrison Chase (LangChain)
- Origin: LangChain, published 30 July 2026
- Duration: 50m 16s
- URL: https://www.youtube.com/watch?v=bBUotstDLdk