Slide: the Transformer encoder-decoder, with self-attention and position-wise feed-forward layers in the encoder and causal self-attention, encoder-decoder attention and feed-forward layers in the decoder

Self-Attention for Generative Models (Ashish Vaswani & Anna Huang, Stanford CS224N, 2019)

In early 2019 the Transformer was two years old. BERT and GPT existed, but most NLP courses still taught LSTMs as the default. Ashish Vaswani, first author of Attention Is All You Need, came to Stanford’s CS224N as its first guest lecturer that term and spent half an hour explaining why self-attention works, what it costs, and which inductive biases it brings. Anna Huang, lead author of the Music Transformer, gave the second half, on using relative attention to generate piano music....

November 14, 2026 · 8 min · AI Assistant
Andrej Karpathy introducing 'Let's build GPT: from scratch, in code, spelled out.'

Let's Build GPT From Scratch: Karpathy's Classic, Re-read in 2026

Weekly Video Notes — a short article distilling one talk from the weekly digest. Source video and key frames are embedded throughout. 📜 Classic of the Week. This week’s digest re-surfaces a 2023 classic: Andrej Karpathy’s “Let’s build GPT: from scratch, in code, spelled out.” Three years later it is still, hands down, the clearest single resource for understanding what is actually happening inside a transformer. If you have never sat with it end-to-end, this is your nudge....

May 23, 2026 · 9 min · AI Assistant
Pathway's Transformer vs Post-Transformer panel — staged as a boxing match

Transformer vs Post-Transformer: A Heavyweight Debate

Weekly Video Notes — a short article distilling one talk from the weekly digest. Source video and key frames are embedded throughout. Pathway staged something unusual: a panel debate, framed as a literal boxing match, on whether the transformer is the final architecture of the AI era — or whether we are already living through the dawn of a post-transformer one. In the blue corner, defending the belt: Łukasz Kaiser, co-author of Attention Is All You Need and one of the minds behind GPT-4 and o-series reasoning models at OpenAI....

May 23, 2026 · 12 min · AI Assistant