Jason Lopatecki, founder of Arize, opens by noting that his own team’s first agent “frankly sucked” — and that everything they’ve built since is downstream of debugging that failure in production. His AI Engineer talk lays out a concrete pattern for agents that repair themselves: production signal (traces, evals, human labels) feeds a second agent that opens pull requests against the first agent’s own prompts, tools, and skills. It’s the operational shape of the “self-improving system” idea, minus the hand-waving.

Why observability is the substrate

Traditional observability was for humans clicking through Grafana. Agentic observability is for other agents — the consumer of a trace is now a debugging agent, not an SRE. That reframes the whole stack.

Traces as first-class input for debugging agents

Every AI system today emits far more signal than it ever did as a stateless microservice: full traces, tool-call arguments, intermediate reasoning tokens, eval scores, cost per span. Lopatecki’s argument is that this signal is now the training data for the next iteration of the agent — not just diagnostic exhaust. If you’re not persisting it, you can’t close the loop.

The reality of “self-improving” in 2026

Lopatecki is careful not to over-promise. Today’s self-improving agents don’t autonomously merge to main. What they do:

  1. Detect — an eval regression, a customer complaint, or a trace anomaly triggers investigation.
  2. Diagnose — a debugging agent pulls the offending trace, related historical traces, and the current prompt/skill definition.
  3. Propose — it drafts a patch: a prompt tweak, a new skill, a tool description edit, a new eval case.
  4. Guard — it runs the patch against a saved eval set (regression suite) before opening the PR.
  5. Human merges — engineer reviews the diff and the eval delta, then approves.

The current shape of agent self-improvement: propose, gate, human merge

The interesting shift is that step 5 is now the bottleneck. The bot can generate ten patches an hour; a human can only meaningfully review a few.

Case study — a financial trading agent

The concrete example he walks through is a trading agent Arise built on top of Alex (their own in-product agent). When the trading agent misbehaved — hallucinated an instrument, misread a chart — the failure showed up as a specific span in a specific trace. The debug loop:

Trading agent trace and failure mode

  • Bad span → cluster similar spans across production → build a mini-eval set from them
  • Ask the debug agent: “what change would make these spans pass?”
  • Debug agent proposes a skill edit (e.g. “always call get_market_context before place_order”)
  • Test on the eval set → if pass rate goes up without regression elsewhere → open PR

The framework the debug agent operates on is skills + repos + pre-baked tools. Skills are versioned, reusable capability bundles; repos are the codebases where the changes land; pre-baked tools are the observability platform’s own primitives (fetch trace, cluster spans, run eval).

Skills, repos, and PRs — the loop artifact

Where this is going

Lopatecki’s forward-looking bet is that the ratio of “agent work per human review” keeps climbing, and that the human’s job shifts from writing to guiding and approving. The observability platform stops being a dashboard and becomes an agent’s IDE — the environment where debug agents plan, execute, and ship fixes.

He’s explicit that we’re not there yet: the guard rails around what the debug agent is allowed to modify, how big a diff it can propose, and what the human sees are all still being invented. But the shape of the loop — production signal → mini eval → proposed patch → gated PR — is settling into a recognizable pattern across serious teams.

Key takeaways

  1. Traces are the training data for the next iteration. If you can’t replay a failure, you can’t fix it — persist everything with enough context to reconstruct the agent’s state.
  2. Build mini eval sets from clusters of failed spans, not from hand-crafted synthetic cases. The bugs you saw are the bugs you need to prevent.
  3. The debug agent is a separate agent from the product agent — it has its own tools (fetch trace, run eval, propose diff) and its own guardrails.
  4. PRs are the artifact of self-improvement, not autonomous merges. The human review is the safety valve.
  5. Skills are the unit of change. Editing a skill is safer than editing raw prompts because skills carry their own tests and metadata.
  6. Human review time is the new bottleneck. Invest in tooling that shrinks the review, not just tooling that generates more patches.
  7. Observability is becoming an agent runtime, not a human dashboard — plan platform choices accordingly.

Source