New Study Reveals How AI Models Learn to Create Images from Limited Data
In recent years, the rapid advancement of AI has enabled machines to generate stunningly realistic images, often indistinguishable from those created by humans. However, the inner workings of these AI models have largely remained a mystery. A fundamental question has puzzled researchers: are these models truly “learning” to create novel images, or are they simply memorizing and recombining elements from their training data? Or, in more technical terms: do they embed a density model of images from which they sample? This question is particularly pertinent given the “curse of dimensionality,” a phenomenon in which the amount of data needed to learn a complex function grows exponentially with the number of input dimensions.
A groundbreaking study by researchers at CDS has shed light on this question. The team, led by Zahra Kadkhodaie, a PhD student at CDS, along with Florentin Guth, CDS researcher, Eero Simoncelli, NYU Professor of Neural Science, Mathematics, Data Science, and Psychology, and Stéphane Mallat, Professor at Collège de France and École normale supérieure, found that AI models known as diffusion models transition from just memorizing specific examples in the low data regime to learning the “true” density of the images they are trained on with enough data, effectively overcoming the curse of dimensionality. Their findings were published in the paper “Generalization in diffusion models arises from geometry-adaptive harmonic representations” at the International Conference on Learning Representations (ICLR).
To arrive at this conclusion, the researchers conducted a clever experiment. They trained two identical neural networks on separate sets of images, with no overlap between the two datasets. Surprisingly, when the models were given the same random input, they generated the same novel image, despite having been trained on different data. “This finding was remarkable because not only were the generated images new and not present in either of the training sets, but it also meant that the two models had learned exactly the same underlying function,” Kadkhodaie explained. This suggests that the models had learned the same fundamental “rules” governing the structure of the images, independent of the specific examples they were trained on.
But how do these models manage to learn such complex rules from a limited set of examples, seemingly defying the curse of dimensionality? The researchers discovered that the AI models have an implicit bias towards learning patterns that efficiently represent the key features of images, such as edges, contours, and textures. “The models have a built-in preference for learning these geometry-adaptive harmonic bases, which allows them to capture the essence of an image without needing to see every possible variation,” Kadkhodaie said.
The implications of this research extend far beyond creating images. By understanding how AI models learn to generate new content, researchers can develop more efficient and effective AI systems for a wide range of applications, from designing new materials to assisting in scientific discoveries.
Moreover, the study’s findings could also help us better understand how the human brain processes and makes sense of the visual world. “Just as AI models learn to fill in missing information based on their prior knowledge, our brains may use similar principles to construct a complete picture of our surroundings from the limited information captured by our eyes,” Kadkhodaie suggested.
As Kadkhodaie and her colleagues continue to unravel the secrets of what she calls the “fundamental science of deep learning,” their work contributes to the understanding of the underlying mechanisms of AI models through scientific hypothesis building and experimentation.
“This research is really just the beginning,” Kadkhodaie said. “We’re excited to keep exploring these ideas and see where they take us.”
By Stephen Thomas
