Agent “skills” — reusable folders of instructions, scripts, and assets that a model loads on demand — have quietly become the packaging unit of the agent ecosystem. Philipp Schmid opens this AI Engineer talk with a brutal statistic from Skills Bench v1.1: of 50,000+ published skills, almost none have evals. Most were AI‑written and never tested. And because agents are non‑deterministic, without evals you have no way to tell whether a failing task is your skill’s fault, the model’s fault, or just noise.

The talk is a 21‑minute practitioner playbook for closing that gap: what skills actually are, why they fail silently, eight rules for writing ones that work, and — most importantly — the smallest possible eval harness that lets you iterate on a skill the way you’d iterate on any other piece of production code.

The two kinds of skills, and why they fail differently

Capability skills vs. preference skills

Schmid draws a sharp line between the two categories that dominate real deployments:

  • Capability skills teach a model to do something it can’t already do (call an obscure API, hit an internal service, use a proprietary format). As foundation models get smarter, capability skills naturally retire themselves — the base model eventually learns to do it. Evals are what tell you when it’s safe to delete one.
  • Preference skills encode taste and organizational convention: “use Tailwind here”, “our components live in apps/web/components”, “never call the v1 endpoint”. Foundation models will never learn these — they’re private to you — so preference skills are permanent infrastructure. They especially need evals, because a broken preference skill drifts your whole codebase silently.

The framing matters because it dictates eval strategy: capability skills need pass/fail correctness checks, preference skills need behavioral assertions (“did it pick Tailwind?”).

Does the format actually work?

Skills Bench results — human-written skills win

Skills Bench v1.1 evaluated model performance on real tasks with and without skills loaded. The headline: human‑written skills beat AI‑written skills, and both beat no skill at all — but the margin on AI‑written skills is thin enough that if you skip evals, you cannot tell whether adding your skill is helping or hurting.

Two rules of thumb from Schmid:

If your skill file is above 500 lines, you should definitely look at splitting it. Progressive disclosure only works if the top-level description stays cheap.

Skills are triggered two ways: model‑invoked (the model reads your description and decides to load the body) and user‑invoked (an explicit /skill reference). Model‑invoked is where most bugs live, because the description is the entire load‑bearing surface — get it wrong and the skill silently never fires.

Eight rules for writing skills that fire and behave

Eight tips for writing good skills

Schmid’s checklist, distilled:

  1. Explain the why and how in the description. “Use this skill when working on a React project with the Interactions API” beats “React helper”.
  2. Keep descriptions concise. They live in the model’s context on every turn. Long descriptions inflate cost and reduce the retrieval signal.
  3. Give the model exploration hints for multi-branch skills (e.g. “AWS reference here, Azure reference there”) rather than dumping both inline.
  4. Do not encode fixed workflows in skills — write a script instead. If step 1 → step 2 → step 3 is always the same, you’re wasting tokens asking the model to re‑derive it. Skills should express goals and constraints, not procedures.
  5. Include negative cases. Everyone documents when to use the skill; almost no one documents when not to. Without negative examples, the model over-triggers.
  6. Test with 10–20 prompts before shipping, including no-ops — instructions the skill should ignore.
  7. Look at real trajectories. Ask a coding agent to trawl your own trajectory logs and surface the skills you actually use vs. the ones that just sit there.
  8. Run ablation evals — always with and without the skill loaded. This is the only way to know if a skill is helping, hurting, or ready to retire.

The minimum viable eval harness

Small eval harness for skills

The talk’s most useful section is a concrete walkthrough of how Schmid’s team built a skill eval for Gemini + the Interactions API. The structure is deliberately boring:

Skill eval structure — prompt, expected checks, agent runner

Each eval is:

  • a prompt (what the user would type),
  • a set of expected checks (assertions on the output),
  • a Python runner that invokes the coding agent with the skill loaded,
  • and — crucially — an isolated workspace per run so evals can’t leak context between each other or “cheat” by reading state from a previous run.

Most checks are cheap deterministic regex/AST matches. LLM‑as‑judge is a fallback for outputs that need semantic scoring, not the default. This keeps the eval loop fast enough to gate every change.

You can only change the skill if it improves the eval score, or you add new evals to cover the change.

That single rule — commit only if evals improve or grow — turns skill authoring from vibes-based prompting into eval‑driven development.

Eval-driven skill development, in a loop

Eval-driven development loop for skills

The workflow Schmid recommends:

  1. Draft the skill.
  2. Write 10–20 evals covering happy paths, edge cases, and no‑ops.
  3. Run with and without the skill loaded (the ablation).
  4. Iterate until the skill‑loaded score beats the skill‑off score by a meaningful margin.
  5. When a new model releases, re‑run — if the base model now matches your skill‑loaded score, retire the skill and delete it. Keep the evals though; they become regression tests.
  6. Run isolated: fresh workspace per eval, no shared state.

The whole talk is a corrective against the current default — hand-writing (or generating) skills, dropping them into a repo, and hoping. Evals turn skills into artifacts you can trust in production.

Key takeaways

  1. Skills without evals are technical debt disguised as capability — you cannot tell if they help, hurt, or are silently no-ops.
  2. Capability skills should retire themselves; preference skills are forever. Design evals accordingly.
  3. 500 lines is a smell. If a skill body is that long, split it and use progressive disclosure.
  4. Keep the description short and rich in when to use and when not to use — model-invoked skills live and die on this text.
  5. Fixed workflows belong in scripts, not skills. Skills express goals; scripts express procedures.
  6. Always run ablations — with and without the skill loaded — as the sole valid signal of usefulness.
  7. Isolate every eval run. Shared workspace state produces false passes (“context cheating”).
  8. Cheap deterministic checks first, LLM-as-judge only when necessary. This is what keeps the loop fast enough to be useful.
  9. Re-evaluate on every model release. New base models retire old skills. Evals tell you which ones.

Source