Using Data Science to Understand Music Cognition

Supported by the Moore Sloan Data Science Grant, two Steinhardt professors team up to explore how the brain responds to complex auditory stimuli in real-time

The buildup before a beat-drop. A long crescendo. Rising pitch. Repetition. Dissonance. These are some familiar examples of musical tension, a broad term that musical theorists associate most closely with changes in loudness, melodic contour, tempo, and harmony.

But how exactly do our brains perceive and respond to musical tension?

A new study by two NYU researchers, Morwaread Farbood, Associate Professor of Music Technology, and Marc Scott, Professor of Applied Statistics and CDS Affiliated Faculty, aims to answer this question. Funded by a seed grant from the Moore-Sloan Data Science Environment at CDS, the researchers will investigate the correlation between neural and behavioral responses to musical tension. Their study will be a novel intersection of data science, music cognition, and signal processing.

Farbood and Scott plan to collect neural and behavioral data from twenty music listeners in real time.

As the listeners hear samples of Western pop, classical music, and non-Western music, they will record their own judgments of musical tension levels with a digital slider. Their recorded judgments of increases and decreases in musical tension will comprise the behavioral data. To collect the neural response data, the researchers will track listeners’ signal processes with electroencephalograms (EEGs).

The data from listener judgments will be evaluated with a predictive music cognition model used in Farbood’s past research. The model can predict the general tension response of listeners based on quantitative changes in musical features (i.e. loudness, tempo, harmony, pitch). By comparing the model to the actual behavioral response data, the researchers can identify how listeners adjust their ratings to different musical styles and how much the behavioral data varies from the model.

But this study’s most substantial contribution to data science will involve developing new models to isolate EEG data related to musical response and correlating that data with the listeners’ behavioral data. With mathematical models that link both neural and behavioral data from music listeners, the researchers will shed light on how our minds perceive music in real time.

By Paul Oliver