Meet NYU’s newest creation: TIM, an AI driven virtual assistant

When you hear the words “virtual assistant”, the first thing that probably comes to mind is Siri or Alexa. They are easily accessible applications on portable devices to search the internet and complete simple tasks like sending a text message or turning on a lamp. However, virtual assistants on mobile devices are often limited to using speech and text cues as indicators and cannot provide advanced assistance with tasks.

To address this problem, the Defense Advanced Research Projects Agency established the Perceptually-enabled Task Guidance program, which aims to develop artificial intelligence technologies to assist with performing complicated tasks. A team of NYU faculty and students received a $5 million contract to develop Transparent, Interpretable, and Multimodal Personal Assistant (TIM), an artificial intelligence technology that can provide immediate audio and visual feedback.

Technology like this has tremendous implications for real-world use, whether you’re a medic walking through an extensive medical procedure or a chef perfecting a complicated baking technique.

But how exactly will this all work? The team at NYU plans to use Microsoft’s Hololens 2 augmented reality system as its hardware platform with its array of cameras, microphones, lidar scanners, and inertial measurement unit sensors. With these sensors, the virtual assistant will process large amounts of data to understand its environment and test the model accordingly.

The project will be led by Claudio Silva, Professor of Data Science, Computer Science, and Engineering at NYU Tandon and NYU CDS. He will be working with 14 NYU faculty and students, including CDS faculty Kyunghyun Cho and He He.

For more information, see DARPA’s website.

By Keerthana Manivasakan