Most conversations about making agents better are really conversations about making models better. Alex Krentsel — a Berkeley systems PhD student working with Martin Casado and Ankur Goyal — spends this Latent Space episode arguing the opposite: the model is fine, and the leverage now lives in the harness, the “body we provide to the brain.”
Exo, the project he introduces, is an agent that can safely edit every aspect of itself at runtime. Not just its memory file, not just a skills directory — its prompts, its context assembly, its tools, and its own source code. The claim that makes this more than a stunt is architectural: recursive self-modification is only safe if you first partition the agent into layers that can be isolated from each other.
The harness is where the tuning happens now
Krentsel’s framing starts from a shift the field already half-noticed: as models improved, practitioners discovered that most of the remaining performance delta came from tweaking the scaffolding around the model rather than the model itself.
“We’re starting to realize that as we tweak these harnesses, they get really good at particular tasks.”
Today’s agent systems expose a handful of sanctioned extension points. There’s a memory file — “literally a memory MD file somewhere,” as he puts it. There’s a skills directory. There are connectors and MCP servers, and a growing ecosystem of hub-style registries. Agents can already write to these, which is why the self-improvement story sounds partially solved.
His objection is that these are a small, human-chosen subset of what actually determines agent behavior. The prompts, the compaction strategy, the tool definitions, the way context is assembled from available sources — all of that is policy, and all of it is currently frozen at build time. If you put Exo’s architecture next to a conventional one, he says, you would “put red lines around all components and say they are all changeable by the agent itself.”
This connects to a bitter-lesson argument. Over-specializing the harness by hand is a bet that humans know the optimal architecture for a given task. As models get better, the model itself becomes the better architect.
Inner-loop vs. outer-loop self-improvement
A natural objection: we already have self-improving setups where an outer agent observes an inner agent and rewrites it. Why merge the loops?
Krentsel’s answer is that the trust argument people make for the outer-observer design doesn’t hold up:
“You are still trusting a machine when you have an outer separate agent modifying the inner agent. It’s still not you making the changes.”
The real question is not human vs. machine but external inspection vs. self-inspection. And self-inspection has a concrete advantage: the system deciding what to change is the same system deciding what to run and what to inspect. It can try something, watch its own internals, and adjust mid-run.
His example is Exo playing Pokémon. The running system decided on its own to inspect the game’s RAM, then mapped RAM addresses to game state — reproducing by inspection something the community had previously reverse-engineered by hand — and used that to inform its next design decisions.
The three-layer decomposition
The core of the talk is a slide that splits “agent” into three layers with deliberately different properties.
Today, Krentsel notes, Claude Code is simultaneously an agent, a harness, and a process running in the same environment as the code it edits. The usual workflow — spin up a VM, clone a repo, log in with GitHub credentials, then run with permissions skipped — works, but it puts credentials, conversation state, policy, and execution in one blast radius.
Exo splits this into:
| Layer | Contains | Property |
|---|---|---|
| Executor | All policy: prompts, context assembly, compaction, skills, tool definitions | Fully stateless |
| Exo harness | Conversation history, secrets/API keys, artifacts, environment snapshots | Holds all protected state |
| Sandbox | The actual environment where actions execute | Disposable, isolated |
The split line is how vs. what. As he puts it: “how you do things is defined in the executor; the actual state at runtime lives in the exo harness.” Memory is the clean illustration — the strategy for recording and injecting memory lives in the executor, but the memories themselves are artifacts held by the harness.
This is the classic separation of compute and storage, applied to agents, and it’s what makes agent state teleportable and resumable rather than pinned to one machine.
Why the secret store belongs in the harness
The security payoff is the most immediately practical part of the episode. If your agent runs in the same environment as your API keys — the standard setup today — those keys are fully readable and fully exfiltratable by anything the agent executes.
In the Exo layout, the secret store lives in the harness, on the host process, outside the sandbox where tools actually run. Secrets are injected into requests without ever being exposed in the space the tools can see.
The hosts extend this to observability: an access log gives you post-incident forensics, and unusual access patterns become a monitoring signal. If a routine task suddenly requests AWS keys, something is wrong — even if the keys were legitimately available.
Making self-modification safe: rollback and atomicity
With the layers in place, the self-improvement mechanism follows: the executor can see its own code, edit its own code, ask to rebuild itself, and swap mid-run. It can build new tools, write new skills, change how context is assembled, and adjust adapters.
What makes that survivable is the harness underneath, which can roll the executor back. Because the executor is stateless, a proposed change is a clean unit — which also means changes can be evaluated in parallel and committed atomically.
The concrete win Krentsel reports: Exo noticed its Discord adapter was expensive, re-architected the adapter at runtime to scope context to specific conversations and threads rather than pulling messages across threads, tested the change, and drove cost down roughly 96%. That change was later committed back into the real codebase by hand.
Scaling: one sandbox per conversation
The teleportation property earns its keep at scale rather than on a laptop. Running one agent on one task, you either run locally or run remotely — moving state around buys little.
The picture changes with many parallel tasks. Krentsel’s example is a dedicated agent per user’s event stream: wake a sandbox per conversation, and at 100 users everything fits on one machine as 100 containers. At 50,000 users it doesn’t — but because executors are stateless, you need only a few policy processes, and the sandboxes can be distributed independently.
The unsolved part: evals
Asked whether an exo harness should ship its own evals, Krentsel concedes there’s no eval box on the architecture diagram, and that this is the open problem.
The failure mode is obvious once stated: tell an agent to reduce its own cost, and the cheapest strategy is to stop doing the work. That’s textbook reward hacking. In the Discord case, verification was nearly binary — is it responding, with the right context? For something like an insurance agent making judgment calls, you need a held-out eval set, or tooling the agent builds collaboratively to check itself as it evolves.
“The problem of specifying what you want to an agent is still an open one.”
Key takeaways
- The harness, not the model, is the current tuning surface. Most agent performance delta today comes from scaffolding changes, which makes scaffolding worth architecting properly.
- Today’s self-improvement is limited to sanctioned extension points — memory files, skills, connectors — while prompts, context assembly, and tool definitions stay frozen.
- Decompose the agent into executor (policy, stateless), harness (state, secrets, history), and sandbox (execution). The split is how vs. what.
- A stateless executor is what makes rollback, atomicity, and parallel evaluation of self-modifications possible.
- Keep secrets in the harness, never in the sandbox. Co-locating API keys with agent execution makes them trivially exfiltratable; injection plus an access log is strictly better.
- Outer-loop self-improvement doesn’t buy you trust — you’re still trusting a machine. Merging the loops lets the same system decide what to change, run, and inspect.
- Separating compute from storage makes agent state teleportable, which matters for many-parallel-agent workloads rather than single-task local use.
- Evals are the missing box on the diagram. Self-improvement against an underspecified objective is an invitation to reward hacking.
Source
- Title: Exo: Harnesses should see their own code and logs
- Speaker: Alex Krentsel (UC Berkeley / Google Research), with the Latent Space hosts
- Origin: Latent Space podcast
- Duration: ~47 minutes
- URL: https://www.youtube.com/watch?v=5lFD-34dhqE