Stanford CS329A: Self-Improving AI Agents — Course Overview

Stanford’s CS329A is one of the few graduate courses aimed squarely at the thing practitioners are actually building right now: agents that improve themselves. This first lecture is the map — a compressed tour from GPT-3-era scaling laws to the agentic loop inside Claude Code, delivered by two instructors who worked on the models in question.

Akanksha Chowdhery is an adjunct professor at Stanford and researcher at Reflection AI; Azalia Mirhoseini is an assistant professor in the CS department who worked on Gemini at Google DeepMind and on Claude at Anthropic. They met at Google Brain. The lecture is worth watching not for novelty but for sequencing — it lays out how each scaling regime ran into its own ceiling and what replaced it.

Act one: parameters, data, compute

The opening is the familiar three-panel scaling-law chart: increase compute, increase dataset size, or increase parameter count, and test loss falls predictably. From BERT’s 340M through GPT-2’s 1.5B, GPT-3’s 175B, and PaLM’s 540B to GPT-4’s estimated trillions, the model-size curve grew exponentially from 2018 to roughly 2024.

Bigger models bought three things: better benchmark performance, few-shot learning (no per-domain fine-tuning needed — just show the template), and emergent behaviors that simply do not exist in smaller models.

The canonical emergent behavior is chain of thought. Given the tennis-ball arithmetic problem, a one-shot example with the reasoning shown (“Roger started with five, two cans of three is six, 5+6 is 11”) outperforms an example showing only the answer. Crucially, it only works above a size threshold: LaMDA and GPT at ~7–8B parameters get nothing from chain-of-thought prompting; larger models leverage it immediately. That threshold effect is precisely why labs keep pushing scale — not just for the predictable loss curve, but for the unpredictable capabilities.

Act two: the ChatGPT recipe

Pre-training is the easy step — next-token prediction over everything. What made ChatGPT (1M users in 5 days) a leap over GPT-3 was the fine-tuning stack layered on top:

1. High-quality next-token fine-tuning — books, curated essays, data companies pay hundreds of millions of dollars for.
2. Instruction tuning — instruction/question/answer triples, historically a blend of human-authored templates and synthetic data, teaching the model to follow and answer.
3. RLHF — instead of supervised labels, humans rate pairs of model outputs, those ratings train a reward model, and the reward model steers the LLM’s parameters. Rewards can be decomposed and weighted: correctness, helpfulness, specificity, harmlessness.

None of this is new to a 2026 practitioner, but the framing matters for what follows: all three operate on the weights, before inference.

Act three: inference is a frontier too

The pivot in the lecture — and in the field around 2024–25 — is that you can buy capability at inference time without touching a single parameter.

Mirhoseini’s lab demonstrated this with Large Language Monkeys (named for the infinite monkey theorem). Take a small model, sample it not once but up to 10,000 times per problem, and use a verifier — unit tests, for instance — to select a correct answer. Llama 3 8B and 7B models, well below GPT-4o at one sample, exceed it once sampling is scaled.

“It seems like the model already knows a whole lot more than what you get out of them when you just ask them once.”

For some problems only three or four of the 10,000 samples were correct — which is the whole point. The capability was latent; the sampling budget surfaced it. Latency is less punishing than it looks because parallel samples are, well, parallel; the real tradeoff is cost.

Then DeepSeek (December 2024), the o1 series, and Gemini thinking models closed the loop. Where Large Language Monkeys showed log-linear scaling in coverage (pass@k), o1 showed log-linear scaling in pass@1 against test-time compute alone. The model learned which of its many possible reasoning chains is the correct one.

The self-improving flywheel falls out of this: test-time scaling generates enormous volumes of high-quality synthetic traces on verifiable problems (math with known answers, code with tests), and those traces become the fine-tuning set for the next model. Train-time and test-time scaling stop being separate regimes.

Inspecting o1-style traces, the instructors identify recurring structure — problem analysis, task decomposition, self-evaluation against feedback (running tests, using a calculator), self-correction, and backtracking to alternative proposals. Whether these are emergent or trained is, they concede, genuinely unclear: “It’s a bit of both.”

Act four: from LLMs to agents

The transition is defined crisply. A chatbot answers. An agent is given a goal, plans steps, interacts with an environment, uses feedback to correct itself, and decides when it is done — or reports that it cannot be.

In practice, most production systems are not open-ended loops but hand-constructed graphs. The lecture enumerates the component vocabulary — LLM calls, verifiers, critics/judges (LLM-as-judge), tool calls, search — and the orchestration patterns that combine them: prompt chaining, routing by complexity, parallelization with aggregation (Deep Research), and orchestrator/manager patterns where a central LLM plans and dispatches.

Coding agents are the sharpest instance. The agent interacts with the computer through the terminal: navigate the repo, search files, view and edit lines, execute commands, read output, decide the next edit. The instructors are direct about the timeline — this loop “was not quite reliable last year, and it’s just starting to get reliable” now. What changed is not the paradigm but more capable models plus RL with verifiable rewards.

The domains where this lands hardest are the repetitive ones: code migrations, version upgrades, restructuring, data-engineering ETL and warehouse migrations, unit tests. Also customer support (transcription, knowledge assist, smart replies, call summarization) and long-form research synthesis.

At the speculative edge: research agents as AI scientists, assisting with idea generation, experiment iteration, and paper writeup. The instructors make an interesting defense of hallucination-prone brainstorming — having read the entire web, these models propose directions outside what a domain-trained researcher would consider.

The bottleneck: the generator–verifier gap

The most valuable exchange in the lecture comes from student Q&A. Chowdhery names the constraint directly: models can generate plausible reasoning traces cheaply, but knowing which ones are useful requires a feedback loop.

In verifiable domains — math, code, rule-based tasks — verifiers are cheap and self-improvement compounds. In creative writing, human feedback becomes the bottleneck and the flywheel stalls. Robust verification, not generation, is what limits how far the loop can run. The course dedicates an entire lecture to verifiers, and Mirhoseini’s lab works on combining them.

Two other honest admissions worth noting. Models “like their own traces more” — even traces from a stronger model are less effective for fine-tuning than self-generated ones. And on whether the gains come from RL or from diverse pre-training data: “I don’t think there’s a single point of consensus at this point in time. Both processes help.”

Logistics, briefly

Second time the course has run, with an added third homework. Course website at cs329a.stanford.edu; grading is 50% homeworks, 50% project. Projects run in teams of two to four with API credits provided, and acceptable shapes include new evaluation datasets or benchmarks, agentic-system reliability studies, or hill-climbing an existing benchmark — explicitly not survey papers or “just an app.” Students published papers out of last year’s projects. Guest lectures come from frontier labs, covering post-training evolution, multimodal agents, and robotics. Lectures go up on YouTube.

Key takeaways

1. Scaling laws bought predictable loss reduction and unpredictable emergent capability. Chain-of-thought only works above a parameter threshold — a large part of why labs keep pushing scale.
2. The ChatGPT delta over GPT-3 was the post-training stack, not size: high-quality SFT, instruction tuning, then RLHF via a learned reward model.
3. Inference is an independent scaling axis. Repeated sampling plus a verifier lets an 8B model beat GPT-4o’s single-sample performance on hard math and code benchmarks.
4. o1-style models moved log-linear scaling from pass@k to pass@1 — the model learned to identify its own correct reasoning chain, without any parameter-count increase.
5. The self-improvement flywheel is test-time compute feeding train-time data. Verifiable domains generate their own high-quality fine-tuning corpus.
6. Agents differ from chatbots by having a goal, an environment, a feedback loop, and a stopping condition — plus memory and tools.
7. Most production “agents” are hand-built graphs, not open loops: prompt chaining, routing, parallelize-and-aggregate, orchestrator/worker, evaluator-judge.
8. Verification is the bottleneck for self-improvement. The generator–verifier gap is why coding and math compound while creative domains do not.

Source

• Talk: Stanford CS329A: Self-Improving AI Agents — Part 1, Course Overview
• Speakers: Akanksha Chowdhery (Stanford / Reflection AI) and Azalia Mirhoseini (Stanford CS; formerly Google DeepMind, Anthropic)
• Origin: Stanford Online, Fall 2026 quarter
• Duration: 1:09:42
• URL: https://www.youtube.com/watch?v=6YnLB0XbTnI