Shaun Abram
Technology and Leadership Blog
Research paper summary: Attention is all you need
Attention is all you need is an era defining paper from eight researchers who all worked for Google at the time. The title is a play on the Beatles song, Love is all you need.
The paper addresses some of the shortcomings of other common approaches in Machine Learning (such as recurrent and convolutional layers), in favor of a new architecture, the Transformer. It is based on an approach called self-attention where the model can associate the current token (or word) to any other token in the document, regardless of it’s position. This makes the model so much better at capturing associations and meaning across documents. Further, the Transformer architecture is able to to this processing in parallel, making it faster that preceding models.
I don’t have a Machine Learning background, so this paper was a tough read for me. My interpretation reflects that and I have relied on (and quoted from) other interpretations to help me, including:
- “The Illustrated Transformer” (github.io) by Jay Alammar, which is the most digestible interpretation I could find, and
- This summary from the University of Notre Dame (although exact authorship unclear)
Still, the original is short (16 pages) and is so influential that it was worth reading directly; it came to define the modern era of AI and of LLMs (Large Language Models).
Neural networks
The paper starts by talking about how typically models at that time (2017) were based on “complex recurrent or convolutional neural networks”
Recurrent Neural Networks (RNNs)
Recurrent Neural Networks (wikipedia.org), or RNNs, process sequential data, such as text, in order, one step at a time.
However by the time the process reaches the end of a long sentence, early information has often faded or been overwritten. This was a known problem with RNNs called the is the “vanishing gradient” problem. Variants of RNNs were created (LSTMs and GRUs) to improve the problem, but didn’t fully solve it.
An analogy is reading a book but only being allowed to keep a small notepad of notes as you go, erasing and rewriting as you move forward.
Additionally, as the paper states, the “inherently sequential nature precludes parallelization within training examples, which becomes critical”. In other words, RNNs process words one at a time, with no ability to parallelize, and that becomes a problem (i.e. slow) with larger documents.
Convolutional neural networks (CNNs)
Convolutional Neural Networks (CNNs) are a type of artificial neural network originally designed for analyzing images. They avoid the vanishing gradient problem (by using back propagation, which to be honest, I don’t understand!), and they were adapted for text. But because they only have a limited “window”, capturing relationships between distant words is still difficult.
Self-attention
The paper goes on to suggest a new approach that avoids the issues experienced with RNNs and CNNs, specifically, attention mechanisms.
We propose a new simple network architecture, the Transformer,
based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
Attention mechanisms were not new at the time (“Attention mechanisms have become an integral part of compelling sequence modeling”), but were typically used along side RNNs or CNNS. Relying on attention mechanism so heavily was the novel approach here.
What are “Attention mechanisms”?
“Self-attention, sometimes called intra-attention is an attention mechanism relating different positions of a single sequence in order to compute a representation of the sequence”
What does that mean? Above we discussed how RNNs have a small window of context (“a small notepad of notes”). With self-attention, instead of that small context, the model has the ability to look back at any and all the words as needed.
Jay Alammar uses this example sentence:
“The animal didn’t cross the street because it was too tired.”
Humans intuitively know that “it” refers to the animal. This is much harder for a model.
When the model is processing the word “it”, self-attention allows it to associate “it” with “animal”.
As the model processes each word using attention, it looks at other words for clues that can help lead to a better encoding for this word. It decides which other words are most relevant right now, weighting them accordingly.
And it is self-attention that the Transformer model depends on:
We propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output.
The Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence aligned RNNs or convolution.
How self-attention works
The Transformer architecture uses a particular form of self attention called Scaled Dot-Product Attention. But honestly, that is where the paper got very complex for me. In fact I found all of section 3 Model Architecture hard to follow. But Jay Alammar has a good go at explaining it in his Self Attention in Detail section.
While encoding the word “it”, part of the attention mechanism focuses on “The Animal”, and baked a part of its representation into the encoding of “it”.
This is done using three vectors: a Query vector, a Key vector, and a Value vector. These vectors are created by multiplying the embedding by three matrices that we trained during the training process.
Again, I found the mechanics difficult to understand, and perhaps to really understand, I would need to play with a TensorFlow implementation to see the model being created taking shape.
And faster too
As well as being able to relate any token in a sequence to any other token, the Transformer architecture is also able perform the attention function in parallel by using a mechanism called “multi-headed” attention which allows the model to “perform the attention function in parallel”. As well as performing work in parallel, each unit of work has it’s own memory, with the data being combined at the end for the result.
The surprising breakthrough
And this novel approach of relying heavily on attention mechanisms and parallelization provided a breakthrough:
Experiments on two machine translation tasks show these models to
be superior in quality while being more parallelizable and requiring significantly less time to train. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.
I like this interpretation in the University of Notre Dame paper:
The Transformer model reimagined how machines might “understand” language, not by mimicking human speech step-by-step, but by treating language as a spatial field of meaning, where connections can be drawn freely and flexibly. It moved us closer to the dream of AI that can reason, translate, summarize, and even create.
And while the goal of the paper was to improve translation efficiency (it focused on English-to-German and English-to-French translation tasks), an unexpected phenomenon emerged:
the models didn’t just learn to predict text—they began to show signs of reasoning. Transformers began handling analogies, completing logic puzzles, and making inferences that seemed more like thinking than memorization. This surprised even the researchers. It turns out that the very process of trying to predict the next word, over and over again, forces the model to internalize patterns in language that mirror human reasoning.
For those less familiar with the technical side of AI: this would be like teaching a child to complete nursery rhymes and accidentally discovering, ten years later, that they can write essays on Shakespeare. No one expected it—but it made modern AI possible.
Beautiful!
Tags: AI, architecture, artificialintelligence, google, large-language-models, llm, machine-learning, neural-networks
Subscribe to RSS Feed