Incredible Alumni: CDS capstone project by Harlan Hutton, Jenna Eubank, and Harshitha Palegar accepted to NeurIPS machine learning and the physical sciences workshop 2022
The project applies machine learning to astronomical image coaddition
Harlan Hutton, Jenna Eubank, and Harshitha Palegar have taken data science to the stars with their paper “Astronomical Image Coaddition with Bundle-Adjusting Radiance Fields”, which was accepted to the NeurIPS 2022 Machine Learning and the Physical Sciences workshop held in New Orleans last December. The three graduates from the CDS masters program initially started the research as their capstone project with CDS Affiliated Professor Dr. Shirley Ho and stayed on as guest researchers at the Flatiron Institute Center for Computational Astrophysics to complete the work.
The research explores machine learning methods in image coaddition, a process used by astronomers to combine multiple images into a single higher-resolution image. The paper proposes a deep learning method to combine, de-noise, and remove obstructions from observations of cosmological objects with various adjustments available for aspects like resolution and noise level. While image coaddition is essential to observable astronomy, these tasks are not currently available through one succinct process.
Since graduating from CDS, Harshitha has gone on to work as a Data Scientist at Warner Bros. Discovery, Jenna as the VP of Data Services at Insider, and Harlan as a Software Engineer at Google. CDS spoke with Harlan about the project, deep learning methods in the field of astronomy, and advice for current CDS students.
Congrats on your paper being accepted into the NeurIPS 2022 Machine Learning and the Physical Sciences workshop. Can you talk a bit about how this project began as well as how you and your teammates became involved?
Thank you! Harshitha Palegar, Jenna Eubank, Zafir Momin, and I were matched with a project studying turbulence with Dr. Shirley Ho at the Flatiron Institute. On our first day, Dr. Ho came to us with a completely unrelated proposal based on recent groundbreaking research using neural radiance fields (NeRF) to reconstruct 3D scenes from a set of 2D images. She had the idea to explore if these methods could be used to combine images of astronomical objects from different telescopes to create a single, higher-quality image (this is not currently possible in astronomy!). Although we were intimidated at first by the ambiguity and our admitted lack of expertise in both computer vision and astrophysics, we were also intrigued by the prospect of a completely novel research topic. Thus, what became a year and a half of radiance fields and star clusters was born!
Where do you see this work evolving next and what questions or areas of exploration has it left you with?
Our paper is meant to show the amount of promise that lies at the intersection of machine learning and astronomical image coaddition. Ideally, we’d love to see another capstone take over where we left off and add more scientific rigor in terms of quantifiable results and formal comparison to existing coaddition methods.
We also began exploring the application of our model to satellite images with the idea that a deep learning method could learn to denoise images that are majority noise, like a set of images of a forest fire where most of the images are full of smoke coverage. We’d love to see a group explore that more — wouldn’t it be so cool and useful if we could reconstruct a real-time scene of a forest fire from satellite imagery?
You mentioned the promise at the intersection of these two fields. In what ways can the field of astronomy benefit from more deep learning methods?
The current methods for image coaddition in astronomy are well researched and optimized — albeit rigid and disjoint. Using bundle-adjusting radiance fields allows for flexibility and for all the coaddition steps to live under one roof, but it is very far from outperforming existing methods. However, if this deep learning method ever does become successful, the images we see of star clusters and galaxies could come from all sorts of different telescopes, and we would be able to throw out less data given the relaxed constraints on noise and seeing levels. Overall, the images could become much richer in information.
Do you have any advice for current CDS students who might be thinking about taking on a project like this?
Our biggest piece of advice would be to not let a project’s domain stop you from pursuing the research in which you’re interested! Even the word ‘astrophysics’ itself is intimidating and we let the imposter syndrome get to us at times. It’s important to remember that no one can be an expert at everything, and people who are experts at what you aren’t are almost always very happy to help fill in the gaps in your domain knowledge.
Another one of our takeaways was how emotionally taxing research can be! We invested months into working on a codebase that we ended up having to completely scrap. It can be very draining to start over, and we had many days when we didn’t think we’d have anything to show for all the time and effort. Those hours we put into the failed code made the second iteration go a little smoother, then the third iteration, until we finally had a good understanding of the issues and changes we needed to make. In our experience, that was the point where we started finding some success. There’s a dance in there between perseverance and knowing when to walk away and start over.
by Meryl Phair
