Weekly Paper Notes — one of the top picks from the 2026-08-08 CS paper digest. Area: Systems / Networking.
Authors: Sarah McClure (UC Berkeley), Tegan Wilson (Northeastern), Brad Karp (UCL / Google), Michael Mitzenmacher (Harvard), Sylvia Ratnasamy (UC Berkeley), Scott Shenker (UC Berkeley / ICSI), Minlan Yu (Harvard) arXiv: 2608.01707 · PDF
TL;DR
Every large ML training system sits on one of two interconnect families: the fat-tree Clos that GPUs inherited from datacenter networking, or the torus that TPUs inherited from HPC. Practitioners have strong opinions about which is better and almost no analytical basis for them. This paper supplies the basis. The authors derive closed-form completion-time bounds for the three collectives that actually carry ML training traffic — AllReduce (data parallelism), AllGather (tensor parallelism), and AlltoAll (expert parallelism / MoE) — on both topologies, then extend those bounds to cover link failures, job placement, switch multicast, hybrid scale-up/scale-out designs, and cost per port. They finish by plugging in real parameters from a 462B-parameter Megatron-LM configuration and DeepSeek V3.
The headline result is not “Clos wins.” It’s more interesting than that: neither topology dominates across all collectives, and the crossover point is a function of node count, message size, and the ratio of per-link bandwidths rather than either bandwidth in isolation. But once you layer on realistic link-bandwidth ratios (TPU links are roughly 2× GPU links today, giving r_b ≈ 0.5), single-link failures, and the reality that a real training job runs three or four parallelisms simultaneously on one network, the Clos comes out ahead in most cases — and, critically, is far more forgiving. The torus can be tuned to beat it, but only with a placement that is correct for one parallelism and actively harmful to the others.
What problem is the paper actually attacking?
The Clos and the torus both predate ML by decades, and each arrived in its current position by inheritance rather than by design for this workload.
The fat-tree Clos [Al-Fares et al., Greenberg et al.] became the datacenter default because it offers flexible bandwidth provisioning, abundant path diversity, and scale-out to large host counts using switches of bounded radix. GPUs happened to be installed in datacenter hosts, so GPU clusters got the Clos essentially for free. The torus came from the other direction: it was the interconnect of choice for parallel scientific machines, where tightly synchronized nearest-neighbour computation maps naturally onto a toroidal mesh, and Google carried it into the TPU line.
There is an appealing intuition that the torus should be better for ML: modern training stacks compose several distinct parallelisms (tensor, data, pipeline, expert), each of which propagates values along a linear chain of dependencies, and a multi-dimensional torus lets you lay one parallelism along each dimension. DeepSpeed’s 3D parallelism diagram practically looks like a torus.
What the literature was missing, as the authors point out, is any analytical comparison that accounts for placement of distinct parallelisms and for degradation under link failure. Those two factors turn out to be exactly where the two topologies diverge most. The gap partly reflects the historical balkanization of the datacenter-networking and HPC-networking communities — two groups optimizing for workloads with very different characteristics, who now find themselves optimizing for the same one.
The motivating observation is disarmingly simple: ML training’s network behaviour reduces almost entirely to three collectives. If you can bound those three on each topology, you have bounded the workload.
The mechanism: closed-form collective completion times
The model is deliberately spare. Per-link latency α, per-byte link bandwidth cost β, per-byte reduction time γ, message size m, torus dimensionality k_t, Clos switch radix k_c, n nodes participating in the collective out of N total. Defaults are drawn from real hardware: α = 1 µs, β = 400 Gbps, m = 100 MB, k_t = 3, k_c = 128.
From these the paper derives best-known-algorithm completion times for each collective on each topology. For AllReduce on a Clos, log₂(n)(hα_c + β_c·m + γ·m); on a torus, (k_t·n^(1/k_t)/2)(α_t + β_t·m) + log₂(n^(1/k))·k·γ·m. For AllGather on a Clos, hα_c + β_c·m(n−1); on a torus the bandwidth term shrinks by a factor of 2k_t, because a torus node has 2k_t access links while a Clos node has exactly one.
That 2k_t factor on AllGather is the paper’s cleanest insight, and it explains the one place where the torus genuinely and permanently wins. AllGather is bound by access bandwidth — how fast a single node can pull n−1 messages in through its own links — not by aggregate network bandwidth. A torus node has six links in a 3-D configuration; a Clos node has one. No amount of bisection bandwidth in the spine fixes that. Correspondingly the tie-line for AllGather in the ratio plot sits at r_b ≈ 1/(2k_t) = 1/6, meaning a torus could run at one-sixth the per-link bandwidth of a Clos and still match it on AllGather.
AlltoAll goes the other way. The Clos’s full bisection bandwidth gives it asymptotically better bandwidth scaling by a factor of n^(1/k_t)/8 and better latency scaling (constant h·α versus k_t·n^(1/k_t)/2·α). AllReduce is latency-dominated — bound by how fast a reduction can reach every node — and the Clos’s log₂(n) tree beats the torus’s linear-in-dimension traversal at scale.
Plugging in the actual bandwidth ratio the industry ships (r_b = 0.5, from five years of published TPU and GPU per-link numbers) resolves the ambiguity: AllReduce is faster on Clos, AlltoAll is faster on torus only below roughly 50 nodes, and AllGather is always faster on torus. Message size barely perturbs this; only below ~500 KB does the ordering flip for AllReduce, where the torus’s smaller diameter starts to matter more than bandwidth.
Why the practical claim holds up: failures and placement
The static comparison is a tie-ish. The tiebreakers are operational.
Failures. At scale, link failures are inevitable, and the paper derives the additive CCT penalty for a single bidirectional link failure on each topology. The Clos absorbs it: path diversity means a failed link is rerouted with a bounded additive cost expressible in terms of rack size R = k_c/2 and pod size P = k_c²/4. The torus does not absorb it. With only 2k_t links per node and no path diversity to speak of, a failed link forces traffic onto a longer route through the ring, and the resulting increase in collective completion time can run into the hundreds or thousands of percent depending on collective and scale.
Placement. In a non-oversubscribed Clos, placement barely matters: bandwidth cost is placement-invariant and latency cost varies by at most an additive 4α_c, the maximum hop difference between any two node pairs. In a torus, placement changes both latency and bandwidth cost. The typical strategy — one parallelism per dimension — yields max distance ½·N^(1/k_t) between group members. A locality-optimized placement that tiles the torus with √N smaller 2-D grids cuts that to 2N^(1/4), an asymptotic improvement, but it necessarily doubles the distance for the other parallelism, because the second group must draw one member from each of the first groups.
Whether that trade is worth it depends on which collective you’re optimizing. If both parallelisms run AllReduce, locality optimization is strictly worse. If the optimized parallelism is AlltoAll — the one collective where inter-member distance drives both the latency and the bandwidth term — it asymptotically wins. A random placement, by contrast, costs you a factor of k_t in latency for everything and another factor of k_t in bandwidth for AlltoAll.
The takeaway is the asymmetry, not the winner: the torus offers tuning headroom the Clos doesn’t have, but it also offers failure modes the Clos doesn’t have. On a Clos, a bad placement costs you 4 µs. On a torus, a bad placement costs you a multiplicative factor.
Results: the LLM case studies
The paper closes by applying the models to two real configurations. For the Megatron-LM 462B example (TP=8, PP=16, DP=48; message sizes 132 MB for TP, 18.9 MB for PP, 7.2 GB for DP) and DeepSeek V3 (671B; TP=1, PP=16, DP=2, EP=64; 28 MB for EP), they compute per-collective CCT on an appropriately shaped torus (8×16×48 and 2×16×32 respectively) and on a k_c=128 Clos with standard placement heuristics — TP and EP placed for highest locality within a rack, then DP, then PP.
The Clos achieves lower or equal CCT for most collectives in both models. The exception is, predictably, AllGather, where the torus’s 6× access-bandwidth advantage carries the day at equal per-link bandwidth. Since AllGather is tensor parallelism, and tensor parallelism is the parallelism you most want to keep off the critical path anyway, the practical guidance follows directly: unless AllGather is your bottleneck, the Clos is the better choice.
The paper also examines switch multicast and hybrid scale-up/scale-out designs. Multicast helps only AllGather (and AllGather-based AllReduce), because AlltoAll flows are all unique data and gain nothing from replication. And multicast only saves transmissions, not the best-achievable CCT, since on both topologies AllGather is already bound by node access bandwidth rather than by longest-path transmission. That is a genuinely useful negative result for anyone tempted to buy in-network multicast as a training accelerator.
Why this matters
This is a paper of the kind we get too few of: it takes a design question that is currently settled by institutional inheritance and vendor allegiance, and settles it with derivations instead. The immediate practical value is a set of formulas an interconnect architect can actually evaluate for their own model shape and hardware generation, rather than having to run a full simulation campaign.
The deeper value is the framing. By showing that the topology comparison hinges on access bandwidth per node for AllGather, bisection for AlltoAll, and diameter for AllReduce, the paper gives you a way to reason about any future topology — NVLink meshes, UALink fabrics, optical circuit switches — without re-deriving from scratch. Ask which of the three quantities the design improves, then ask which collective your model is bottlenecked on.
The obvious follow-up is the one the authors gesture at: hybrid topologies are already the norm (one fabric for scale-up, another for scale-out), and the paper’s Table 6 begins the Clos+mesh and Clos+torus analysis but doesn’t push it to the same depth. Given that essentially every frontier training cluster shipping today is a hybrid, that’s where the next paper should go. Power and physical layout — explicitly excluded here for tractability — are the other missing axis, and they are the reason torus deployments persist despite the failure-resilience penalty.
Read alongside
- Al-Fares, Loukissas, Vahdat, A Scalable, Commodity Data Center Network Architecture (SIGCOMM 2008) — the Clos fat-tree as datacenter default.
- Jouppi et al., TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning (ISCA 2023) — the torus in production, with optical reconfiguration as the resilience answer.
- Shoeybi et al., Megatron-LM — the parallelism decomposition whose message sizes drive the case study.
- DeepSeek-AI, DeepSeek-V3 Technical Report — the MoE configuration that makes AlltoAll a first-class concern.
- Thakur, Rabenseifner, Gropp, Optimization of Collective Communication Operations in MPICH (2005) — the collective algorithms the bounds are built on.
Links
📄 arXiv abstract · 📄 PDF
Part of the Weekly CS Paper Digest series. Summary written from a close read of the preprint; figures cropped from the arXiv PDF and reproduced here under fair use for educational commentary.