Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks
Brenden Lake contributes to research investigating how machines handle compositionality
The ability to acquire language is often taken for granted. From the utterance of a first word, to the eventual sentences of early childhood, language quickly comes to encode our thoughts and enable our interpersonal communications. Language is so fundamental, it seems almost instinctual. Yet, for computers, absorbing natural language is incredibly difficult. Where a child could easily interpret the meaning of a novel phrase, “to run slowly,” knowing both the definitions of “to run” and “slowly,” a computer struggles to make these same, seemingly intuitive, connections. It is important to understand how the mind handles compositionality in language and thought in order to attempt to reproduce these abilities in machines.
In their recent publication, Brenden M. Lake, Assistant Professor of Psychology and Data Science, and Facebook AI Researcher, João Loula, of École Polytechnique and Facebook AI Research, and Marco Baroni of Facebook AI Research, explore recurrent neural networks’ challenges in compositional abilities and generalization. Previously, Lake and Baroni produced another research paper, which also spoke to the inadequate systematic compositionality skills of neural networks. Researchers conjecture that this problem is rooted in the need for large training datasets. They challenged recurrent neural networks to combine a novel verb, “dax,” with known components such as “again” and “twice.” Even having seen input such as “jump twice” and “jump again,” RNNs fail to understand “dax twice.”
Responding to concerns that this merely confirmed known issues with one-shot learning (the ability to learn from one or few examples), Lake et al.’s new publication deals with combining known words in unseen patterns. The new study strengthens the original findings that RNNs struggle with compositional generalization. RNNs are good at generalizing from many examples of a familiar pattern, but fail when generalization requires new application of compositional rules. An example of this might be struggling to infer the meaning of “around right,” given that the meanings of both “around” and “right.” In order to study compositionality in seq2seq neural network models, researchers re-purposed the SCAN dataset introduced in Lake & Baroni.
Before jumping in, a quick tour of a seq2seq neural network model:
Most modern machine translation systems rest on a framework wherein two neural networks communicate with each other. The first, an encoder neural network, creates a unique context vector for a given sentence. It accomplishes this by reading a word, creating a vector, reading in the next word, and updating that original vector. The final product is a vector that represents the sentence. The second neural network, rather intuitively, is called a decoder neural network. The task of this neural network is to output the resulting phrase in a different language. To accomplish this, the network is initialized with the context representation vector developed in the first network, then outputs a word in, let’s say, German. It then feeds this word back into the system, reviews the context it had, and proceeds to the next word. The process continues until the translation of the original sentence is complete.
Lake et al. test compositionality when the model has seen ample examples of each word, but not in combination with one another. In Lake et al., a model is taught the meaning of the word, “around.” It is taught about the meaning of both “right” and “left,” but is only taught “around” in the context of left. Upon asking the model “Can you jump around to the right?”, it fails. The model knows the meaning of “around,” having seen it in all possible contexts from left. It knows the meaning of “right.” A human would find it easy to extrapolate from the symmetry between left and right, and use “around” in the novel context of “right.” However, Lake et al.’s new research proves that the problem was not solely introducing a new verb like “dax”; there is a fundamental error with compositional learning. Fortifying the conclusions in Lake and Baroni, researchers found that RNNs still struggle with systematic compositionality in this adapted task.
By Sabrina de Silva
