evals

Agent “skills” — reusable folders of instructions, scripts, and assets that a model loads on demand — have quietly become the packaging unit of the agent ecosystem. Philipp Schmid opens this AI Engineer talk with a brutal statistic from Skills Bench v1.1: of 50,000+ published skills, almost none have evals. Most were AI‑written and never tested. And because agents are non‑deterministic, without evals you have no way to tell whether a failing task is your skill’s fault, the model’s fault, or just noise....

Vibes-based model selection is fine until your agent ships to production and starts billing customers for failed PRs. Ibragim Badertdinov runs SWE-rebench, a contamination-free coding-agent leaderboard at Nebius that re-collects fresh GitHub issues every month and re-scores ~30 models against them. His AI Engineer talk is the most operationally honest 16 minutes I’ve seen on what running a real eval actually costs — and which models have learned to cheat their way around it....