Research Spotlight: CDS and Grid AI researchers propose new method of self-supervised learning
In the realm of self-supervised learning, state-of-the-art algorithms can broadly be divided into two groups: contrastive approaches and non-contrastive approaches. But what if there could be a third, unique approach? This is the main question that was answered in a recent publication authored by CDS PhD Student William Falcon, along with researchers at Grid AI and CDS faculty Kyunghyun Cho. The paper, titled “AAVAE: Augmentation-Augmented Variational Autoencoders” introduces the titular augmentation augmented variational autoencoders or “AAVAE.”
The research team looked at quite a few techniques from different domains like vision, NLP and reinforcement learning, and identified that data augmentation was crucial in all of them. With this in mind, the team went back to a somewhat outdated technique, variational autoencoders or VAEs, as a foundation from which to build a new self supervised learning methodology. They identified the KL divergence in VAEs to be a weak point and decided to replace it with a denoising criterion and domain-specific data augmentations. The result of this change is the AAVAE.
After performing experiments to test AAVAE’s effectiveness, the team found that it performed substantially better than its predecessor, the VAE, and even out performed certain contemporary self-supervised learning methods. While it is still not competitive with most contemporary variants, it made significant progress in closing the gap between them and the standard auto-encoding techniques. The paper ultimately concludes that AAVAE proves the potential of autoencoding as an alternative to contrastive and non-contrastive approaches for self supervised learning and advocates for further investigation into the subject.
We at CDS are excited to see data scientists question the field’s preconceived notions and find unlikely solutions to old problems! If you’d like to read more about AAVAE, you can find the research at arxiv.org. You can read more about PhD Student William Falcon and his company Grid AI on the CDS Blog!
