Symile: A Simple, Powerful Way to Help AI Systems Understand Multiple Types of Data Together
When analyzing real-world information, we often need to process many different types of data simultaneously — like interpreting video that combines visuals, audio, and text captions. A new paper from CDS introduces a method that could revolutionize how AI systems handle these complex scenarios.
Courant PhD students Adriel Saporta and Mark Goldstein, CDS Faculty Fellow Aahlad Puli, and CDS Associate Professor of Computer Science and Data Science Rajesh Ranganath have developed Symile, a flexible method that learns to understand relationships between unlimited types of data simultaneously.
Previous work like CLIP, a popular AI method, learns to understand relationships between pairs of data types, like matching images with their text descriptions. CLIP works by maximizing what’s called “mutual information” between two types of data. However, when dealing with more than two types of data, CLIP can only examine relationships between pairs in isolation, missing important patterns that emerge from looking at everything together.
Symile addresses this limitation by targeting “total correlation,” which captures relationships between all types of data simultaneously. This allows Symile, introduced in a paper called “Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities,” to understand complex patterns that only emerge when examining multiple data sources at once. The method is remarkably flexible — it can work even when some types of data are missing, and it requires no specialized architectures or complex fusion models.
“Symile captures strictly more information than CLIP,” said Saporta, who works on multimodal representation learning and AI for health in Ranganath’s lab at the Courant Institute, in a thread on X/Twitter. Plus, “once you compute Symile representations, the original data isn’t needed anymore — giving you efficient modality-specific vectors for any downstream task.”
The researchers demonstrated that Symile could be particularly useful in healthcare, having successfully analyzed patterns between chest X-rays, electrocardiograms, and laboratory tests from over 11,000 hospital admissions. “One of the exciting things about Symile is that it can enable resource-constrained hospitals to take advantage of advances in AI without requiring massive computing power,” said Saporta. “The representations learned by Symile can be shared and used by institutions that don’t have access to large-scale computing infrastructure.”
“In places where they might not have access to all types of medical imaging equipment, Symile can help make predictions using whatever data is available,” explained Puli. “If a hospital doesn’t have X-ray machines but has ECG and lab equipment, they can still benefit from models trained on the complete dataset.”
Beyond healthcare, Symile could also advance robotics applications by helping robots better understand relationships between different types of sensor data. Current robots often struggle with tasks like picking up a green screwdriver — while they might understand “screwdriver” and “green” separately, they have trouble combining these concepts. Symile’s approach to processing multiple types of data simultaneously could help robots better understand these kinds of composite relationships.
In general, Symile would benefit any domain that relies on vector embeddings, including multimodal generative modeling. “Symile can help make predictions using whatever data is available,” explained Puli. “If some types of information are missing, the model can still leverage whatever is present to make effective predictions.”
The team has made both their method and datasets publicly available to encourage further research. By providing a simple but powerful way to analyze multiple types of data together, Symile opens new possibilities across many fields where complex, multimodal data analysis is crucial.
By Stephen Thomas
