Why AI That “Sees” Physics May Need to Look Beyond the Pixels
Physical systems that look chaotic to the eye — water swirling down a drain, fluids mixing at a boundary, particles drifting in a stirred medium — are usually governed by just a handful of simple equations. Flatiron Institute Research Fellow Helen Qu and her collaborators — including CDS Research Scientist Michael McCabe, CDS Senior Research Scientist Shirley Ho, and CDS founding director and Professor Yann LeCun — have found that machine learning models trained to predict the future in an abstract “latent space,” rather than pixel by pixel, do a noticeably better job of recovering physical information from raw observations.
The paper, “Representation Learning for Spatiotemporal Physical Systems,” was presented at the ICLR 2026 Workshop on AI & PDE. It asks a simple question: which kind of model actually learns physics, rather than just learning what physics looks like?
The motivation comes from a longstanding problem with AI emulators of physical processes. Most surrogate models are trained to predict the next frame of a video — the next snapshot of a fluid flow, say — and they tend to drift away from reality sooner than you might expect. “If you make a small mistake at step n, then step n+1 takes in that mistake, and then it continues to roll out, it diverges significantly from reality after a surprisingly low number of steps,” Qu explained.
Qu and her co-authors propose sidestepping the problem. Instead of asking a model to reproduce every pixel, they ask whether it can recover the parameters governing the equations behind a system. The approach builds on Joint Embedding Predictive Architectures, or JEPAs, which co-author LeCun has championed as a path toward AI that builds reliable models of the world. JEPAs predict not the next frame, but a compressed, abstract representation of it.
The team benchmarked four models on three physical systems from The Well, a collection of physics simulations released by the Polymathic AI initiative last year: active matter, Rayleigh-Bénard convection, and shear flow. Two of the models were generic self-supervised methods originally designed for natural images and videos: a custom JEPA the team built for this work, and VideoMAE, which masks and reconstructs pixels. The other two were designed specifically for physics: Multiple Physics Pretraining (MPP), an autoregressive foundation model led by co-author McCabe, and DISCO, a meta-learning operator approach developed by Flatiron Research Fellow Rudy Morel and collaborators.
Across all three physical systems, the results were consistent. The JEPA outperformed VideoMAE, and DISCO beat MPP. Furthermore, the two best performers in each class — JEPA and DISCO — shared a feature that distinguished them from the others: both predict in a latent space rather than at the pixel level. In other words, what they share is the choice not to commit to reconstructing every detail of the input.
“Learning in the latent space gives you the ability to not be bound by very granular, pixel-level correctness,” Qu said.
The team also looked at how performance scaled with fine-tuning data. With only 10 percent of the available training examples for the shear flow task, the JEPA outperformed VideoMAE’s best result on the full dataset.
Qu was careful about the scope of the claim. The work doesn’t establish that latent prediction is the right paradigm for every problem in scientific machine learning. As she put it, the question of what counts as “physically grounded” representations is “totally up for debate. I’m sure there are plenty of people who will say, this is not really a great proxy for physical representation. And, sure. Find me a better one.”
The paper was produced through Polymathic AI, the cross-institutional initiative working on foundation models for science, with co-authors including Alberto Bietti of the Flatiron Institute’s Center for Computational Mathematics and François Lanusse of CNRS and Université Paris-Saclay.
By Stephen Thomas
