https://sparsenotes.com/tags/transformers/ Recent content in transformers on Sparse Notes https://sparsenotes.com/images/og-default.png https://sparsenotes.com/images/og-default.png Hugo -- gohugo.io Sat, 23 May 2026 00:00:00 +0000 https://sparsenotes.com/posts/2026/05/karpathy-lets-build-gpt/ Sat, 23 May 2026 00:00:00 +0000 https://sparsenotes.com/posts/2026/05/karpathy-lets-build-gpt/ Notes on Andrej Karpathy's 2023 classic 'Let's build GPT: from scratch, in code, spelled out.' — tokenization, the bigram baseline, the math of self-attention, multi-head, residual + layernorm, and how the same ~200 lines scale to a real GPT. https://sparsenotes.com/posts/2026/05/transformer-vs-post-transformer/ Sat, 23 May 2026 00:00:00 +0000 https://sparsenotes.com/posts/2026/05/transformer-vs-post-transformer/ Notes from the Pathway-hosted 'Transformer vs Post-Transformer' panel — Łukasz Kaiser defending attention, with Adrian Kosowski (BDH/Pathway), Matthias Lechner (Liquid AI) and Llion Jones (Sakana AI) arguing for what comes next. Scaling laws, latent reasoning, hardware lock-in, benchmarks, and whether transformers themselves will discover their successor.