evals

Most retellings of the modern AI wave start with ChatGPT arriving out of nowhere in November 2022. Ross Taylor has a different vantage point: he shipped a competing language model two weeks earlier, watched it get torn apart in public, and spent the following four years working out exactly why. This talk is the compressed version of that education — half war story, half technical agenda for what comes after the current generation of agents....

Two and a half years after the Devin demo went viral at 13% on SWE-bench, Cognition president Russell Kaplan sits down with Harrison Chase for the most candid accounting yet of what running coding agents at enterprise scale actually costs — and why the industry’s incentives are quietly pointed in the wrong direction. The central claim: a lot of the ecosystem is structurally motivated to get customers to token-max, and the bill is now coming due....

Jason Lopatecki, founder of Arize, opens by noting that his own team’s first agent “frankly sucked” — and that everything they’ve built since is downstream of debugging that failure in production. His AI Engineer talk lays out a concrete pattern for agents that repair themselves: production signal (traces, evals, human labels) feeds a second agent that opens pull requests against the first agent’s own prompts, tools, and skills. It’s the operational shape of the “self-improving system” idea, minus the hand-waving....

Prompt engineering was the primitive of 2023, context engineering owned 2024–2025, and 2026 is settling on a new one: loop engineering — the discipline of designing the feedback loops that surround an agent, not just the agent itself. Sydney Runkle (PM on LangChain’s open-source team) makes the case in this webinar that the durable advantage is never the agent, it’s the loops built around it. Why loops, not agents Runkle opens with a simple framing: a model has some fixed level of intelligence; a harness wrapped around it converts that intelligence into useful work on a specific problem....

Agent “skills” — reusable folders of instructions, scripts, and assets that a model loads on demand — have quietly become the packaging unit of the agent ecosystem. Philipp Schmid opens this AI Engineer talk with a brutal statistic from Skills Bench v1.1: of 50,000+ published skills, almost none have evals. Most were AI‑written and never tested. And because agents are non‑deterministic, without evals you have no way to tell whether a failing task is your skill’s fault, the model’s fault, or just noise....

The Latent Space “Cooking with…” series put OpenAI’s Chief Research Officer Mark Chen in a kitchen and got him to talk through the things research-org leaders rarely say on the record: where scaling laws actually live in 2026, why post-training and RL are the real bottleneck now, how OpenAI structures evals against a moving frontier, and what “AGI” means when you’re inside the org that named the goal. This is one of the higher-signal AI Engineering interviews of the year — partly because Chen is unusually specific, partly because the format (informal, no slides, no PR minder) catches him in mid-thought....

Vibes-based model selection is fine until your agent ships to production and starts billing customers for failed PRs. Ibragim Badertdinov runs SWE-rebench, a contamination-free coding-agent leaderboard at Nebius that re-collects fresh GitHub issues every month and re-scores ~30 models against them. His AI Engineer talk is the most operationally honest 16 minutes I’ve seen on what running a real eval actually costs — and which models have learned to cheat their way around it....