A mathematical mechanism enables AI models to process language at scale, undergirding the current AI revolution. Our columnist asks: Why is attention the key?
Read Later
Introduction
Everything that’s swept the world since the 2022 release of ChatGPT — “the AI economy”; data centers; chatbots acting as best friends, doctors, and mathematicians; talk of existential risk — it all starts, in some sense, with an intern in a University of Montreal lab named Dzmitry Bahdanau.
It was 2014. The 23-year-old Belarusian, who had been coding competitively since he was a kid but hadn’t done academic research before, was asked to help figure out how to get a computer to translate sentences from English to French. Bahdanau was working with artificial neural networks, which had already shown promise in pattern-recognition tasks such as deciphering handwriting and identifying objects in photos. The networks learn to recognize patterns by strengthening and weakening connections between nodes in response to training data. Bahdanau quietly doubted their potential to achieve anything like human-level intelligence, but his adviser, the computer scientist Yoshua Bengio, and the postdoc on the project, Kyunghyun Cho, were believers. And translation was widely seen as a formidable and interesting problem for neural networks to try to solve.
Translation is not as simple as swapping each English word for a French one. Languages have different grammatical structures and are riddled with idioms and connotations, all of which had long stymied attempts to automate the process. You can’t simply implement rules of translation because there are none. But neural networks seemed promising because they learn by example. They can be trained on huge databases of paired English and French sentences to, eventually, absorb the quasi-rules, exceptions, and subtleties.
Results had been mediocre, though; the longer the sentence, the worse neural networks performed. That’s because researchers were trying to cram each sentence’s data into a fixed amount of memory before decoding the sentence in the target language. It was a bottleneck that limited how complex and sophisticated translations could be. The network architecture just wasn’t right.
So Bahdanau tried something different. “It was just obviously the right thing to do — one of these things that happens once in a while in science where it’s just very clear to someone who is not very experienced in the field,” he recalled, speaking to me from Montreal, where he still works in AI. “And it just worked. From the very first attempt.”
Bahdanau’s trick, a mathematical operation that he, Cho, and Bengio referred to as “attention,” enabled their neural network to translate long sentences just as well as short ones. That meant the approach was scalable. Other AI researchers at the time quickly integrated attention into their neural network designs. Then, in 2017, a team at Google devised a new kind of neural network architecture called a transformer that supercharged the attention method and stripped away everything else. The team’s paper, “Attention Is All You Need,” reported that transformers achieved state-of-the-art translation while requiring a fraction of the computation needed to train prior models. The paper, which has since been cited more than 250,000 times, was a watershed moment in AI. Large language models such as ChatGPT and Claude followed.
Given the staggering power that’s been unleashed, it’s worth asking what this attention mechanism is exactly, why it is “all you need,” and why machine translation was the task that unlocked it all. Attention is a way for us to understand how a single AI model can converse with us, diagnose diseases, generate realistic videos, solve long-standing math problems, and more. Is it somehow at the heart of intelligence — artificial and otherwise?
Sentences with pronouns such as “it” in them illustrate the challenge that attention overcomes. Consider: “The animal didn’t cross the street because it was too tired.” And: “The animal didn’t cross the street because it was too wide.” To translate these sentences correctly into French, an AI model must determine that, in the first one, “it” refers to the animal, which is a masculine word in French, so the translation should use that form of the pronoun, il. But in the second sentence, “it” refers to the street, a feminine word that requires a different pronoun, elle. How can the machine figure this out?
Bahdanau remembered doing translation exercises when he was learning English in school. To write each next word of his translated sentence, he’d have to glance all around the source text to figure out what should come next. “Literally you have your source sentence on the left and your writing on the right, and you’ll be, like, turning your head, left-right, left-right,” he said. “That process is what I expressed as a neural mechanism.” The tool of attention that he gave to his neural network was a mathematical way for the network to identify the word or words anywhere in a sentence that would most inform the next word of the translation. In the case of our ambiguous pronoun, “it,” the system navigates the rest of the sentence to determine which words affect its meaning the most.
A transformer consists of many layers of nodes connected to nodes in adjacent layers. When a sentence is fed into a transformer, each word is first converted into a numerical representation called a vector, which we can imagine as an arrow pointing in a certain direction. These vectors pass through the network, and numbers called weights (determined by the model’s training) dictate how they are transformed in each layer. The weights turn each word’s current vector representation into three new vectors specific to that word — a “query,” “key,” and “value” — that each serve a different purpose in the attention operation.
The network performs this operation several times in parallel through systems called attention heads. An attention head compares each word’s query with the keys of all the other words in the sentence. How well these vectors match up — how much they point in the same direction — determines how strongly the value of each other word contributes to a new, updated vector for the word at hand. Information about other words in the sentence literally becomes incorporated into each word’s representation. The system is converting context into math.
When “The animal didn’t cross the street because it was too tired” enters a transformer, the first stack of layers in the network, known as the encoder, uses the attention operation to build up a contextual understanding of the sentence. In each layer, each attention head generates a query (along with a key and value) for “it.” This query vector can be thought of as a request for information that exists elsewhere in the sentence. For instance, in a deep layer (after the network has already accumulated information about words), the query for “it” might be interpretable as: “I need an earlier noun that could plausibly be the thing that is tired.” Meanwhile, the key associated with “animal” might effectively advertise itself as a noun that’s capable of being tired, among many other features. Its key therefore matches the query better than the key for “street.” And that causes the value associated with “animal” to contribute more heavily to the updated representation of “it.”
That new, context-enriched representation of “it” passes into the next layer. There, the network performs attention all over again. Successive rounds give the words increasingly rich representations of themselves.
Experts say it’s often hard to interpret what any given attention head is doing, but collectively they pick up on many kinds of relationships, from the partnership between nouns and verbs to the association between living beings and feelings. Later layers build on associations established by earlier layers, so that by the time the sentence emerges from the encoder, “it” is imbued with something resembling meaning.
After that, the second part of the transformer, the decoder, produces the French translation one word at a time, using attention to draw on the encoder’s numerical representations of the English words as it decides which French word to spit out next. The contextual information encoded in the English sentence helps translate “it” as the masculine il.
Crucially, no one had to tell a transformer that animals get tired and streets may be wide. The neural network discovers such associations during training, by learning from huge numbers of English sentences paired with their French translations. These associations get built into the weights. The network starts out with random weights that produce unhelpful queries, keys, and values, and thus poor translations. But each translation error sends a correction signal backward through the network that slightly alters the weights that produced it. Through trillions of these adjustments, the network eventually generates useful vectors that yield correct translations, even when it processes sentences it has never seen before.
Attention turned out to be the right mathematical operation for linking words and building context — the fundamentals of translation. And when you can translate blocks of text of unlimited size, it seems you can do anything.
A neural network that has learned enough about language to translate sentences from English into French can also complete sentences in either language. Predicting the next word of a translation is, in other words, a subtask of the more general task of “predicting the next word,” period. And that’s what large language models do. They’re trained on the contents of the entire internet, learning which words should attend to which other words in all sorts of contexts. They process and reprocess the whole dataset and try trillions of times to predict the next word in whatever part of the web they happen to be in at any point during training. The connections between neurons in these models are gradually tweaked in a way that produces patterns of attention between words that are ever more useful for predicting the next word in any context — or for producing an appropriate response to a prompt from a human.
From the ability to translate the word “it,” we somehow got a revolution.
“That’s quite a leap, isn’t it?” said Llion Jones, one of the authors of “Attention Is All You Need,” who is now the chief technology officer of Sakana AI.
Jones and others suspect that predicting the next word, which is known as language modeling, is “AI-complete”: a task so difficult that solving it means you have arguably solved AI in general. The reason it may be AI-complete is that this task includes, as a subtask, virtually anything humans do in written form. “The internet has translations,” Jones notes. “So if I said ‘“The animal didn’t cross the street because it was too tired” in French is [blank],’ then in order to complete that sentence, the transformer needs to be able to translate into French. If I say ‘2 + 3 is [blank],’ it needs to do arithmetic. If it sees a bunch of mathematical symbols in a row and wants to know what comes next, it needs to understand high-level mathematics. If I say ‘The president of the United States is [blank],’ it needs to know facts. So being able to predict the next word requires you to be able to do all these things. And because it’s AI-complete, language modeling is kind of the closest definition that we have to intelligence.”
It might not be apt to say that an LLM “knows” or “understands” anything — not in the way that a human does — but that’s just how we’re accustomed to talking about the abilities they display. Regardless, attention is the tool by which artificial neural networks access and convey meaning, association, and context, often well enough to astonish us with their fluency. (For tasks such as advanced mathematics, today’s frontier AI models often have more explicit formal reasoning and proof verification systems layered on top of their language modeling to check their work.)
But what about human intelligence? Might our brains employ a mechanism similar to the mathematical operation Bahdanau invented? After all, we, too, do something akin to “producing the best possible next word” when we speak, write, and even think. Perhaps our form of intelligence is tantamount to language modeling, too?
Cognitive scientists have begun to study this question — a new version of the ancient question of how language relates to thought. To be sure, the success of LLMs shows how much information about the world is encoded in the statistical patterns of language alone. The question is: When we master language, does intelligence come for free?
Recent research persuades me that human thought is not purely linguistic (and most other animals’ intelligence is not linguistic at all): When we are thinking, more brain regions light up than just the specialized networks in the cerebral cortex that are associated with language. Language is how we communicate about the world, but our thoughts seem to draw on sensory impressions, physical and social experiences, memories, and reasoning abilities that exist at least partly outside the brain’s language networks. So language modeling seems to be merely a powerful component of human cognition, not its equivalent or source.
As for how we do “it” — how I instantly understand that it’s the animal rather than the street that is tired — no one yet knows. Jones, for his part, now thinks that attention is notall you need to truly master language modeling. After all, the brain evidently accomplishes this task far more efficiently than transformers do. We converse fluently without having read the whole internet even once, and without burning as much energy. “The current paradigm won’t solve this [efficiency] problem,” Jones said. “You seem to have to jump to something very, very different.”
Regardless of what our own neural mechanisms might be, it makes sense to me in hindsight that solving translation unlocked AI. Whether you’re human or machine, the ability to translate effectively means you have understood, in some sense. Now the machines understand so much — effectively, at least — that we hardly know what to think anymore.
Also in Computer Science
As AI Closed In on ‘Unique Games’ Proof, Researchers Raced to Beat the Machines
October 7, 2026
Read Later
The Four-Color Theorem Gets a Rare New Proof
The Four-Color Theorem Gets a Rare New Proof
September 10, 2026
Read Later
Live from ICM 2026: What Is Math For in the Age of AI?
Live from ICM 2026: What Is Math For in the Age of AI?
Next article
How Does Life Unfold? A Landscape Metaphor Comes Into Its Own.
