People usually treat cost and quality as a trade-off: pay for a bigger model or more thinking, or save money and accept worse answers. Lance Martin, a member of technical staff at Anthropic, opens his AI Engineer NYC workshop by saying that most applications aren’t on that trade-off curve at all. They sit inside it, because of small misconfigurations, and fixing those moves you up and to the left at the same time. The talk covers five of them, each paired with a Claude Code command that automates the fix, and then spends forty minutes on audience questions, mostly about evals.
Most applications sit inside the frontier
The framing slide plots cost per task against quality. Fable, Opus, Sonnet and Haiku trace a Pareto frontier, and the typical application is a point below and to the right of it. Martin calls the gap a “free” win: lower cost with no drop in quality, which you get by fixing configuration rather than buying more model. The rest of the talk is a list of where that gap usually comes from.
Theme 1: prompts accumulate debt
Prompts pick up habits the way code does, and many of them were written for models that needed hand-holding. Martin’s list: “think step by step”, verification rituals (“double-check your work”), thoroughness boosters (“be maximally thorough”), prescribed scratchpads, and stuffed worked examples. Current frontier models already decide when to think and already check their own work, so these instructions mostly add tokens. The slide groups the habits by what they do to you: some tax cost, some break accuracy, and some (anti-laziness tool boosters, dated API settings) just break.
Contradictory rules are the sharpest case. Older models with weaker instruction following would quietly ignore one of two conflicting rules, so the prompt still worked. Newer models try to honour both and get confused; the slide reports one planted contradictory pair costing about nine accuracy points.
The fix is a Claude Code skill, run as /checkup prompt-audit for your Claude Code configuration or /claude-api prompt-audit for application code. Both call the same underlying skill, which scans the working directory for these anti-patterns and removes them. Martin’s demo is a synthetic 50-ticket customer-support eval. He injects each anti-pattern into a prompt written for Opus 4.8, simulates a migration to Opus 5.5, and then runs the audit. The migration alone moves the point a little left and up; the audit moves it further in both directions, because the model stops spending tokens on rituals it doesn’t need.
Theme 2: use the prompt cache, and don’t break it
Agents run for many turns and resend most of their context each time. Cached tokens are billed at roughly a tenth of the base rate (check the pricing page for the current figure) and are faster to serve. The catch is that caching is a strict prefix match, so a single changed byte early in the prompt invalidates everything after it. Martin’s ordering advice: system instructions and tools first (cached globally), then CLAUDE.md and memory (cached per project), then session state, then the per-turn messages. The Claude console shows your cache hit rate, and he says the Claude Code team tracks theirs “very religiously” because it drives the economics of the product.
/claude-api cost-optimize checks a codebase for cache-breaking prompt layouts, and also looks at effort settings and prompts. Martin and colleagues ran it against the default harnesses of several internal benchmarks. The slide’s numbers, with score changes all inside the confidence intervals shown:
| Benchmark | Cost before → after | Change | Score change |
|---|---|---|---|
| tau2-bench (retail) | $48.68 → $13.20 | −72.9% | +2.6 pp |
| LegalBench | $38.46 → $12.74 | −66.9% | −0.8 pp |
| OfficeQA Pro | $252.14 → $71.26 | −71.7% | +1.5 pp |
| SWE-bench Verified | $14.42 → $10.92 | −24.3% | +1.1 pp |
Theme 3: build evals a model can climb
Claude is good at self-correction, so given a well-built eval it can improve against that eval on its own. Martin lists four properties he checks for, none specific to Anthropic. Scores should rise as you step up the model. They should rise as you increase thinking effort. There should be headroom left at the frontier. Run-to-run variance should be low.
The failure he sees “all the time” is an eval whose last 20% is ambiguous or mis-graded. It looks like headroom, but no model can score there, so you can burn a lot of tokens hill-climbing on something that is actually saturated. /claude-api build-eval walks you through building an eval from raw traces and checks the grader with you. It doesn’t care which tracing product the data comes from. During Q&A he says Hamel Husain’s public dogfooding of the skill convinced him it should show the user the raw data before analysing it, and that a PR adding a data viewer is open.
His grading advice from the Q&A is concrete. Write a rubric a human could apply. Prefer binary pass/fail checks, several of them if needed, over a 1–10 scale, which he calls “extremely noisy”. Have Sonnet, Opus and Fable all grade with the rubric; if they disagree, the rubric is bad. Run replicates to confirm low variance. Pairwise comparison (“which of these two pages is better?”) is often easier for both humans and models than absolute scoring.
He also warns against filling an eval with the failures of one model. Model capability is jagged, so cases picked from Sonnet 5’s weak spots may simply get solved by the next model, leaving an eval shaped around one model’s blind spots rather than around production.
Theme 4: hill-climb without overfitting
With a good eval, Claude can iterate on a prompt or skill unattended. The main risk is overfitting: the harness grows an OCR tool because a few benchmark tasks need OCR, or the prompt picks up phrasing specific to the eval set, and none of it carries over to production. /claude-api hillclimb borrows a standard idea from statistics. It splits the eval into train and test. The analyser only ever reads train-split failures, proposes one change per round, and is scored on both splits. Train improving while test stays flat is the early warning.
Martin’s own example is the Claude API skill itself, which distils Anthropic’s documentation. He built an eval from the docs, the skill started at 66%, and a hill-climbing loop with access to the skill text and the documentation improved it substantially. He recommends text surfaces (prompts, skills) as the place to start, because they’re small, easy to change and tightly coupled to behaviour. Harness code is possible but harder: diffs are bigger and can be unrelated to the signal you’re measuring. The flip side is that prompts are also the easiest thing to overfit.
You can also point the climber at cost instead of quality: hold performance at parity and search over model, effort and prompt. On the support benchmark it started from Opus 4.8 at high effort (74.4% accuracy), moved to Opus 5 at low effort (better and slightly cheaper), dropped to Sonnet 5 at low effort (much cheaper, a little worse), then tuned the prompt to recover. The net result was about a 90% cost cut with better accuracy. This works even on a saturated eval, since you’re only trying to cut cost.
The heuristic built into the skill, and his general effort advice:
“I like very smart model not trying too hard.”
A stronger model at low effort often uses fewer tokens than a weaker model at high effort, so it ends up both cheaper and better. He suggests starting with Opus 5.5 at low effort and raising it only if needed, the opposite of the common habit of starting at max.
Theme 5: harnesses for multi-day agents
Martin walks through METR-style task-duration charts: Claude Opus 3 managed about four minutes of autonomous work, 2025 models reached an hour or two, and current models are past what METR can measure, at a day or more. At that length reliability and security take over as the main design concerns. Claude Managed Agents follows his “decouple the brain from the hands” principle. The harness and the sandbox are separate processes, and either can die and be replaced. The session log is an append-only object independent of both, so a crashed overnight run can resume. Secrets live in vaults the model can reference but never read. /claude-api managed-agents-onboard sets up a worked example (a scheduled agent that gathers context updates and posts to Slack).
In the Q&A he ties this to the recursive language model (RLM) idea. Because context lives outside the harness and the model queries it through tools, compaction doesn’t lose anything: the full history can always be retrieved later.
Key takeaways
- Before switching models, audit your prompts. Verification rituals, thoroughness boosters and scratchpad scaffolds written for older models now cost tokens and sometimes accuracy, and contradictory rules hurt newer models more than older ones.
- Prompt caching is a strict prefix match. Put stable content (system prompt, tools) first and per-turn content last, and track your cache hit rate as a metric.
- A climbable eval gets better with bigger models and more effort, has headroom left, and has low variance. An ambiguous last 20% means hill-climbing is wasted spend.
- Binary rubric checks, cross-model agreement and replicates give much more reliable LLM-as-judge signals than 1–10 scales.
- Hill-climb with a held-out test split the analyser never sees, and watch for train improving while test stays flat.
- Hill-climbing can target cost at fixed quality, and that works even on a saturated eval.
- Try a stronger model at low effort before a weaker model at high effort; it is often cheaper and better.
- Long-running agents need the session log, harness and sandbox to fail independently, and secrets the model can use without seeing them.
Source
- Talk: “Your Old Prompts Are Making Claude Worse” (slides titled “Unhobbling the frontier”)
- Speaker: Lance Martin, Member of Technical Staff, Anthropic
- Event: AI Engineer NYC, workshop, October 12, 2026
- Duration: 64 minutes (about 24 minutes of slides, then Q&A)
- Video: https://www.youtube.com/watch?v=ys6YQ2yfBpM