Anima Anandkumar spent 83 minutes on Latent Space explaining a specific architectural claim: transformers are the wrong tool for physical simulation, and the reason is not compute budget but a mismatch in what the architecture assumes about its inputs. She has the receipts — her group’s weather model, FourCastNet, is deployed at a national weather agency and runs tens of thousands of times faster than the numerical models it replaces.
Anandkumar led AI research at NVIDIA and is now at Caltech; she recently joined the UN scientific advisory board. The conversation covers neural operators, formal verification of neural networks in Lean, fusion plasma control, and inverse design, but the through-line is one idea: physical data is continuous, and architectures that pretend otherwise inherit a ceiling they can’t train their way out of.
The resolution problem
Standard neural networks take fixed-size inputs and produce fixed-size outputs. Images are 224×224, videos assume a frame size, token sequences have a maximum length. That assumption is harmless for vision and language. It is fatal for physics.
“With the standard neural network, you zoom in and it gets blurry. That’s the end of it, there’s nothing more.”
Neural operators model inputs and outputs as continuous functions rather than grids. The practical payoff is that you can train at one resolution and evaluate at another — feeding in finer-grained observations at inference, or adding physical constraints at a resolution the training data never had. Anandkumar is careful that this is not free super-resolution: a model asked to invent detail will invent wrong detail. What makes it work is that conservation laws and other physical constraints are enforced as loss terms at the finer scale, so the extrapolation is pinned to something real.
The Fourier neural operator specifically buys efficiency. Physical solutions are non-local — solving a PDE amounts to integration over a domain, so a point’s value depends on distant points. A transformer captures that with quadratic attention, which Anandkumar dismisses as arithmetically hopeless at simulation scale: a 3D-plus-time domain with a thousand grid points per dimension gives you hundreds of billions of context positions. The Fourier transform gets non-locality at quasi-linear complexity instead. Nonlinearities between Fourier layers restore the expressivity that a purely spectral method would lose.
The weather result
The group tried weather forecasting in 2021 largely because ERA5 reanalysis data was openly available. Domain experts warned them off — decades of careful bottom-up physics modeling stood in the way.
The model matched traditional forecasts on the phenomena those forecasts handle well, ran tens of thousands of times faster, and fit on a consumer-grade GPU. DeepMind, Huawei and others followed within a year. Anandkumar’s framing of why this matters is not about benchmark numbers: cheap inference means small weather agencies in the global south can run forecasts that previously required a supercomputer.
Atmospheric rivers are the illustrative case — narrow moisture corridors thousands of miles long. Capturing them requires exactly the non-local modeling the Fourier operator provides, and a purely local architecture will miss them structurally rather than statistically.
Where the naive architecture breaks
FourCastNet 3 adds two things that turn out to matter far more than expected. First, spherical geometry.
Most weather models use a standard map projection and implicitly assume the world is a rectangle. Over a two-week forecast this is survivable. Run the same model for months, as climate work requires, and error accumulates fastest at the poles until the rollout diverges. Anandkumar draws a sharp line here: a model that only does short-range weather is “a narrow surrogate.” Asking one architecture to span weather and climate forces you to respect that the earth is a sphere.
Second, ensembles. Extreme weather is where AI forecasting earned public trust — the hurricane Lee landfall call came several days earlier than the traditional models managed. But a single deterministic rollout can’t express that. Ensembles were historically the most expensive part of numerical weather prediction; cheap inference makes them nearly free.
The detail that surprised the interviewer, and is worth repeating: the model is trained on six-hour steps with only a little multi-step fine-tuning, yet remains stable over rollouts spanning months. Each ensemble member independently satisfies the physical constraints; the ensemble average is taken over predictions, not over constraints.
Beyond weather
The same machinery shows up in fusion plasma, where the target is predicting disruption events — the moment plasma touches the containment vessel and the reaction ends.
Every disruption in a real tokamak is an expensive, destructive experiment, so moving that work into simulation is the whole game. Anandkumar’s group is working on tokamaks and stellarators, and increasingly on designing the control system and the simulation together rather than sequentially.
Aerodynamics involves a geometric trick worth knowing about: map a car body to a torus, model on the torus, map back.
And the frontier she is most interested in is inverse design — not simulating a given configuration but asking what configuration is best.
The quantum gate example is instructive because human intuition is genuinely bad at it. A fast forward model plus an optimizer produces designs that are efficient and, importantly, physically valid — the constraint machinery from the simulator carries over.
The data constraint nobody talks about
Anandkumar returns repeatedly to a number: her weather model trained on roughly 50,000 samples. That is nothing next to language model corpora, and in most scientific domains it’s worse, because generating each sample means running an expensive simulation.
This is why the field cannot simply copy the language playbook. With scarce data, inductive biases stop being optional and physics constraints have to be built into the architecture rather than learned. It is also the argument for foundation models in this space — not one model per PDE, but a model that has absorbed enough physical structure to be fine-tuned on a new phenomenon with a few samples instead of a fresh 50,000. The open problem she names is coupled physics: the real world does not decompose into one equation at a time.
She also spends time on TorchLean, a framework for expressing neural networks in Lean so you can formally prove input-output bounds. The motivation is control loops — if you want a neural network in the loop controlling a physical system, “it usually works” is not an acceptable guarantee. Certified robustness bounds, and verification that a network satisfies given physical laws, both fall out of the same framework.
Key takeaways
- Neural operators represent inputs and outputs as continuous functions, so a model trained at one resolution can be evaluated at another — the property that makes them viable for physical data.
- Attention is arithmetically infeasible at simulation scale. A 3D-plus-time domain at a thousand points per dimension yields hundreds of billions of context positions; Fourier transforms capture the same non-locality at quasi-linear cost.
- FourCastNet matched traditional weather models on their strengths, ran tens of thousands of times faster, and fit on a consumer GPU — which democratizes forecasting for agencies that can’t afford supercomputers.
- Spherical geometry is not a refinement. Rectangular-projection models diverge at the poles over long rollouts, which rules them out for climate work regardless of their two-week accuracy.
- Six-hour training steps plus light multi-step fine-tuning produce rollouts stable over months. Anandkumar herself flags this as surprising.
- Cheap inference makes ensemble prediction affordable, and ensembles are what made extreme-weather forecasting credible to the public.
- Scientific ML operates on roughly 50,000 samples where language models operate on trillions of tokens, which forces physics constraints into the architecture instead of leaving them to be learned.
- Inverse design, not simulation, is where Anandkumar sees the largest gap between human intuition and what a fast differentiable forward model plus an optimizer can find.
Source
Why Transformers Hit a Wall the Moment Physics Shows Up — Anima Anandkumar (Caltech), in conversation on the AI for Science segment of Latent Space. 1h 23m. Published 26 August 2026.
https://www.youtube.com/watch?v=79mIutht1f4
Neural operator library and documentation: https://neuraloperator.github.io