zig

Tokenizers are a standard example of code you can’t vectorise: a loop that switches on each character, with data-dependent branches everywhere. Niles Salter has spent more than two years on an accelerated Zig parser, and at Software You Can Love Vancouver 2026 he explains why that reputation is wrong and how to write tokenizers differently. His reframing is to treat each instruction as a hardware-accelerated loop of fixed length. Once you see subtract, count-trailing-zeros and vector compaction that way, most of a tokenizer’s branching moves into a few instructions....