Can AI Learn a Language from a Textbook?
Most of the world’s 7,000 languages will never have enough written text online to train an AI translation system. However, linguists have spent years producing detailed grammars and dictionaries that describe these languages in precise form, and a recent line of research has asked whether today’s LLMs are good enough at following instructions to translate from those descriptions alone, without any direct examples of the language.
NYU Linguistics PhD student Jackson Petty and his collaborators found that the answer, for now, is only partially yes. Their paper, “Evaluating In-Context Translation with Synchronous Context-Free Grammar Transduction,” tests how well current AI models translate between language pairs when given a formal grammar instead of training data, and identifies where they fall short.
The research was conducted with CDS MS student Jaulie Goe and CDS Associate Professor of Linguistics and Data Science Tal Linzen, who directs the Computation and Psycholinguistics Lab at NYU and is Petty’s PhD advisor.
Rather than test on real low-resource languages, where verifying translation quality is costly and the properties that shape a translation — word order, morphological complexity, writing system, grammar size — can’t be varied independently, the team used synchronous context-free grammars, a mathematical formalism from the 1960s, to generate pairs of synthetic languages from scratch. This controlled “petri dish” let them verify translations exactly, vary one of those properties at a time while holding the rest fixed, and guarantee no contamination with the models’ training data. Goe’s contribution was making those synthetic languages feel plausible. She developed a method to generate vocabulary across different writing systems that still looked and sounded like real words, an answer to the natural objection that formal languages are too rough a proxy for natural ones.
“Because we control how these languages are generated, we can do lots of fun things,” Petty said, “like intentionally vary the word order, or intentionally vary the morphological complexity of the languages.”
What makes this more than a curiosity is that word order and morphology are among the real ways natural languages differ from one another. In ordinary data those features always travel together, which makes it hard to say which one is driving a given result. But here they don’t have to: a synthetic language can be held constant in every respect but one, which allows the researchers to watch what that single change does to translation accuracy.
The models tested, GPT-5 and Google’s Gemma 3 series, understood the task and translated short sentences from small grammars with near-perfect accuracy. But performance deteriorated quickly as complexity increased. Grammar size was the biggest factor: accuracy dropped sharply as grammars approached the scale needed to describe a natural language.
Word order turned out not to matter much, but morphology did, with the biggest morphological obstacle being when the target language required verb agreement that the source language didn’t mark. The most serious type of obstacle, though, was orthography, in the form of less-common writing systems: accuracy fell when the language to be translated used Cyrillic, fell further with Hebrew, and hit zero across all conditions with Hebrew script with vowel markings (nikkud), called pointed Hebrew.
The likely explanation for why script matters so much is a conflict between two signals: the explicit instructions in the grammar, and the model’s tendency to produce token sequences that resemble its training data — pointed Hebrew simply looks unlike anything the model has seen before.
“There’s a conflict between what the models are being trained to do really well, which is what most of their data looks like, and the explicit instructions that are provided to them,” Petty said. “And that disconnect doesn’t necessarily go away just with more training on general things.”
The failures the paper identifies point to training deficiencies rather than fundamental limits, and Petty suggested they’re probably closable with focused effort. The synthetic grammar framework could also be turned around to generate low-cost training data for real low-resource translation systems.
By Stephen Thomas
