This research paper explains the attention idea that let a translation model look back at different source words for each word it writes, instead of forcing the whole sentence through one fixed-length vector.

How much of a sentence fits in one vector?

Imagine Ravi is translating a sentence from Hindi to English.

Suppose the source sentence is:

“सीता सुबह बाजार गई और अपने भाई के लिए एक किताब खरीदी।”

A basic sequence-to-sequence model reads the sentence word by word:

सीता → सुबह → बाजार → गई → और → अपने → भाई → के → लिए → एक → किताब → खरीदी

The encoder tries to compress everything it has understood into a single fixed-size vector:

Entire sentence → Encoder → [ Context Vector ]

The decoder then has to rely on that same vector to generate the complete translation:

[ Context Vector ] → Sita → went → to → the → market → in → the → morning → and → bought → a → book → for → her → brother

The problem is that one vector has limited capacity.

For a short sentence such as:

“राम घर गया।” → “Ram went home.”

compressing the meaning into one vector may work reasonably well.

But consider a much longer sentence containing several people, actions, relationships, places, and time references. The encoder must somehow preserve all of that information inside the same fixed-size representation.

Conceptually, the architecture looks like this:

Source sentence

Sita ──► went ──► market ──► bought ──► book ──► brother
   │        │         │          │         │         │
   └────────┴─────────┴──────────┴─────────┴─────────┘
                         │
                         ▼
                ┌─────────────────┐
                │ Context Vector  │
                │   [0.21 ...]    │
                └─────────────────┘
                         │
                         ▼
                     Decoder
                         │
            ┌────────────┼────────────┐
            ▼            ▼            ▼
          Sita          bought       book ...

Every detail of the input has to pass through that narrow context-vector bottleneck.

Bahdanau, Cho, and Bengio argued that this becomes especially problematic for long sentences. As the encoder reads more words, earlier information can become difficult to preserve accurately. The decoder is then expected to recover every important detail from a single compressed representation.

You can think of it like asking Ravi to read a long paragraph, write only one small note about it, throw the paragraph away, and then translate the entire paragraph using only that note.

For short sentences, the note may contain enough information.

For long sentences, important details may disappear.

This limitation motivated a major idea:

Instead of forcing the decoder to remember the whole sentence through one vector, let it look back at different parts of the source sentence whenever it generates a word.

That idea became attention.

The decoder needs different evidence at different moments

Translation is not a process where every output word depends equally on the whole source sentence.

At each step, the decoder may need to focus on a different part of the input.

Consider:

Hindi: “सीता ने एक किताब खरीदी।” English: “Sita bought a book.”

When the decoder is generating:

Sita

the most useful evidence is:

सीता

When it generates:

bought

it should pay more attention to:

खरीदी

And when it generates:

a book

the important source phrase becomes:

एक किताब

We can imagine the decoder changing its focus as it translates:

Source:
सीता      ने      एक      किताब      खरीदी
 ↑                          	         	
 │
 └──────────────► "Sita"

                      ↑
                      └──────────────► "a book"

                                   ↑
                                   └──────────────► "bought"

The important point is that the decoder does not need the same information at every moment.

Even grammatical decisions can depend on nearby words. When choosing an article such as “a”, the decoder may need information about the noun it is about to produce. When choosing a verb, it may need to focus on the action expressed somewhere else in the source sentence.

A single permanent context vector forces all of these different requirements to compete inside the same representation:

                  ┌────────────────────┐
Whole sentence ──►│ One context vector │
                  └─────────┬──────────┘
                            │
              ┌─────────────┼─────────────┐
              ▼             ▼             ▼
           person          verb          noun
           "Sita"        "bought"       "book"

The decoder is effectively asking the same vector several different questions:

Who are we talking about? What happened? What object was involved? Which grammatical form should I use now?

Those questions may require different evidence from the source sentence.

Attention changes the setup.

Instead of asking one vector to permanently summarize everything, the decoder can construct a new context for each output step:

                SOURCE SENTENCE
     ┌────────┬────────┬────────┬────────┐
     ▼        ▼        ▼        ▼        ▼
   Sita      one      book    bought    ...

      │                         │
      │ strong attention        │
      ▼                         │
    "Sita"                      │
                                │
                          strong attention
                                ▼
                             "bought"

                  strong attention
                         │
                         ▼
                      "a book"

So rather than asking:

“Can the whole sentence fit into one vector?”

attention asks a more useful question:

“Which parts of the sentence matter for the word I am generating right now?”

That shift—from one fixed summary to a dynamically chosen context—is the central intuition behind the attention mechanism.

The baseline has one reusable context

The original encoder-decoder does not translate an entire sentence in a single step. It generates the target sentence one word at a time.

For a source sentence (x) and target sentence

$$y=(y_1,y_2,\ldots,y_T),$$

the translation probability is decomposed into a sequence of next-word predictions:

$$ p(y\mid x)

\prod_{t=1}^{T} p(y_t \mid y_1,\ldots,y_{t-1},c) $$

At every decoding step, the model considers:

• the target words it has already generated,

• its current decoder hidden state,

• and the source-sentence representation (c).

The important part is that (c) stays the same throughout the entire translation.

Consider:

Source: “Sita went to Delhi yesterday to meet her brother.”

The encoder reads the complete sentence and produces one representation:

Sita → went → Delhi → yesterday → meet → brother
                         │
                         ▼
                ┌──────────────────┐
                │ Context vector c │
                └──────────────────┘
                         │
          reused at every decoding step
                         │
       ┌─────────────────┼─────────────────┐
       ▼                 ▼                 ▼
     "Sita"            "went"          "Delhi"
       │                 │                 │
       ▼                 ▼                 ▼
   "yesterday"          "to"            "meet" ...

The decoder's internal state changes from one step to the next because it has generated different words.

But its direct representation of the source sentence does not:

$$c_1=c_2=\cdots=c_T=c$$

This is why the architecture can be described as having one reusable context.

The decoder may need very different source information at different moments—perhaps the person's name when generating Sita, the location when generating Delhi, and the temporal expression when generating yesterday—but every one of those decisions begins with the same source summary (c).

Importantly, Bahdanau, Cho, and Bengio were not claiming that a fixed context vector can never represent a sentence successfully.

Their question was more specific.

Earlier encoder-decoder models showed that translation quality tended to deteriorate as source sentences became longer. This raised the possibility that the architecture itself was imposing a difficult constraint:

$$\boxed{ \text{source of arbitrary length} \longrightarrow \text{fixed-width } c \longrightarrow \text{entire translation} }$$

A short sentence and a long sentence are therefore asked to pass through the same-sized representational channel.

"Ram sleeps."
      │
      ▼
 ┌─────────┐
 │    c    │
 └─────────┘


"Sita went to Delhi yesterday to meet her brother
 after finishing her work at the university ..."
      │
      ▼
 ┌─────────┐
 │    c    │
 └─────────┘

The width of the context representation does not grow with the sentence.

So the research question is not:

“Can neural networks ever compress a sentence into one vector?”

It is closer to:

“Does forcing every source sentence into one fixed-length representation make translation unnecessarily difficult, particularly as sentences become longer?”

Bahdanau attention changes exactly this constraint. Instead of requiring the decoder to reuse one permanent source summary, it allows the decoder to construct a different context vector (c_t) at every output step:

$$c \quad\longrightarrow\quad c_1,c_2,\ldots,c_T$$

That small change in notation represents a fundamental architectural change: the decoder no longer has to recover every source detail from the same reusable summary.

One annotation stays beside every source position

Instead of compressing the whole source sentence into one vector, Bahdanau attention keeps a separate representation beside every source position.

The encoder is bidirectional.

A forward recurrent network reads the sentence from left to right:

$$x_1 \rightarrow x_2 \rightarrow \cdots \rightarrow x_{T_x}$$

and produces forward states

$$\overrightarrow{h}_1,\overrightarrow{h}2,\ldots,\overrightarrow{h}{T_x}.$$

At the same time, a backward recurrent network reads the same sentence from right to left:

$$x_{T_x} \rightarrow \cdots \rightarrow x_2 \rightarrow x_1$$

and produces backward states

$$\overleftarrow{h}_1,\overleftarrow{h}2,\ldots,\overleftarrow{h}{T_x}.$$

At each source position (j), the two states are concatenated:

$$h_j= \begin{bmatrix} \overrightarrow{h}_j\ \overleftarrow{h}_j \end{bmatrix}$$

This vector (h_j) is called an annotation.

Consider the sentence:

“Sita bought a book yesterday.”

Forward encoder  ─────────────────────────────►

Sita        bought        a        book       yesterday
 │             │          │          │             │
 ▼             ▼          ▼          ▼             ▼
→h₁           →h₂        →h₃        →h₄           →h₅


◄─────────────────────────────  Backward encoder

Sita        bought        a        book       yesterday
 │             │          │          │             │
 ▼             ▼          ▼          ▼             ▼
←h₁           ←h₂        ←h₃        ←h₄           ←h₅

The two directions are combined at every position:

Sita        bought        a        book       yesterday
 │             │          │          │             │
 ▼             ▼          ▼          ▼             ▼
h₁            h₂         h₃         h₄            h₅

So (h_4), the annotation near “book,” does not represent only the word book.

Its forward component has already seen:

Sita → bought → a → book

while its backward component has seen:

yesterday ← book

Therefore (h_4) contains information about book in its surrounding sentence context.

The important architectural change is that these annotations are not immediately collapsed into one permanent vector.

Instead, the encoder leaves behind a row of representations:

$$h_1,;h_2,;h_3,;\ldots,;h_{T_x}$$

You can think of them as an addressable memory of the source sentence:

Source sentence

 Sita       bought       a        book       yesterday
  │            │         │          │             │
  ▼            ▼         ▼          ▼             ▼
┌────┐       ┌────┐    ┌────┐     ┌────┐        ┌────┐
│ h₁ │       │ h₂ │    │ h₃ │     │ h₄ │        │ h₅ │
└────┘       └────┘    └────┘     └────┘        └────┘
   ▲            ▲         ▲          ▲              ▲
   └────────────┴─────────┴──────────┴──────────────┘
              available to the decoder

The source sentence is therefore no longer sealed away inside a single context vector before decoding begins.

Its internal structure remains accessible.

The decoder can later ask:

“Which of these annotations matters most for the word I am generating now?”

If it is generating a person's name, it may focus strongly on (h_1).

If it is generating the object book, it may focus on (h_4).

If it is generating a time expression, it may focus on (h_5).

So the encoder has changed from:

$$\boxed{ x_1,\ldots,x_{T_x} \rightarrow c }$$

to:

$$\boxed{ x_1,\ldots,x_{T_x} \rightarrow h_1,h_2,\ldots,h_{T_x} }$$

This is the representation that makes attention possible.

The next question is no longer “How do we store the whole sentence in one vector?”

It becomes:

“At decoding step (i), which source annotation (h_j) should receive the most attention?”

This is cross-attention before Transformers

In modern terminology, this mechanism is closest to cross-attention.

The decoder has its own recurrent state, while the encoder has produced a separate collection of source annotations:

$$h_1,h_2,\ldots,h_{T_x}$$

At decoding step (i), the recurrent decoder uses its previous state (s_{i-1}) to ask which source annotations are most relevant.

Conceptually:

             ENCODER MEMORY

Source:   Sita     bought      a      book
           │          │        │        │
           ▼          ▼        ▼        ▼
          h₁         h₂       h₃       h₄
           │          │        │        │
           └──────────┴───┬────┴────────┘
                          │
                          ▼
                 attention mechanism
                          ▲
                          │
                    decoder state
                       sᵢ₋₁
                          │
                          ▼
                    next target word

The decoder is therefore attending across two different sequences:

$$\text{target-side decoder state} \quad\longrightarrow\quad \text{source-side encoder annotations}$$

That is why it resembles what we now call cross-attention.

But it is important not to project the entire Transformer architecture backward onto the model.

Bahdanau attention is still built around recurrent neural networks:

$$\text{Bidirectional RNN encoder} + \text{RNN decoder} + \text{attention}$$

There are no Transformer blocks, no multi-head attention, and no stack of token-to-token self-attention layers.

It is also not self-attention.

In self-attention, elements of a sequence attend to other elements of the same sequence:

$$x_i \rightarrow x_1,x_2,\ldots,x_n$$

Here, the decoder state attends to representations produced by a different sequence:

$$s_{i-1} \rightarrow h_1,h_2,\ldots,h_{T_x}$$

So a useful modern interpretation is:

$$\boxed{ \text{Bahdanau attention} \approx \text{recurrent encoder-decoder cross-attention} }$$

The terminology came later, but the underlying idea is already visible: one representation queries a collection of representations from another sequence.

Each target word receives a fresh weighted view

The decoder does not choose exactly one source annotation and ignore all the others.

Instead, it assigns a weight to every source position.

For target step (i), it first computes an alignment score between the previous decoder state (s_{i-1}) and each encoder annotation (h_j):

$$ e_{ij}

a(s_{i-1},h_j) $$

Here, (a) is a learned alignment model.

The score (e_{ij}) asks roughly:

How relevant is source position (j) to the target word I am about to generate at step (i)?

Suppose the source sentence is:

“Sita bought a book.”

For one decoding step, the model might produce scores resembling:

                     alignment scores

Source        Sita       bought       a       book
               │            │         │        │
               ▼            ▼         ▼        ▼
             eᵢ₁          eᵢ₂       eᵢ₃      eᵢ₄

These raw scores are converted into normalized attention weights using a softmax:

$$ \alpha_{ij}

\frac{\exp(e_{ij})} {\sum_{k=1}^{T_x}\exp(e_{ik})} $$

so that

$$\sum_{j=1}^{T_x}\alpha_{ij}=1.$$

The weights describe how strongly the decoder should use each source annotation.

For example, while producing “book”, the distribution might look conceptually like:

Source word       Attention

Sita                 0.03
bought               0.12
a                    0.20
book                 0.65
                     ────
                     1.00

The decoder then forms a new context vector specifically for this decoding step:

$$ c_i

\sum_{j=1}^{T_x} \alpha_{ij}h_j $$

This is a weighted view of the source sentence.

h₁ ──× αᵢ₁──┐
h₂ ──× αᵢ₂──┤
h₃ ──× αᵢ₃──┼──►  Σ  ──►  cᵢ
h₄ ──× αᵢ₄──┘

The crucial difference from the old encoder-decoder is that (c_i) is not permanent.

At the next decoding step, the decoder state changes. That changes the alignment scores:

$$ e_{i+1,j}

a(s_i,h_j) $$

which changes the attention weights:

$$\alpha_{i+1,j}$$

and therefore produces a different context vector:

$$c_{i+1}.$$

So instead of

$$c,c,c,c,\ldots$$

the decoder receives

$$c_1,c_2,c_3,\ldots,c_{T_y}.$$

Each target word gets its own freshly constructed view of the source.

For a translation such as:

Sita bought a book

the changing focus might look like:

SOURCE
Sita        bought         a        book
 ████         ░             ░         ░
   │
   └────────────────────────────► "Sita"


Sita        bought         a        book
  ░          ████           ░         ░
               │
               └────────────────────► "bought"


Sita        bought         a        book
  ░           ░           ██       ████
                                      │
                                      └────► "a book"

The model therefore no longer asks one vector to answer every translation question.

Instead, every decoding step can ask the source sentence a new question and construct the context it needs for that particular target word.

That is the central move of Bahdanau attention:

$$\boxed{ \text{one fixed source summary} \quad\longrightarrow\quad \text{one weighted source view per target step} }$$

Attention wins the matched neural baselines

The WMT14 English-to-French experiments give the clearest quantitative comparison between the conventional fixed-context encoder-decoder, RNNencdec, and the proposed attention model, RNNsearch.

The authors trained both model families on the same selected parallel corpus and used the same general training procedure. They trained matched versions using sentences up to either 30 words or 50 words.

At the 30-word cutoff,

$$21.50-13.93=\mathbf{7.57}$$

BLEU points separate the attention model from the fixed-context baseline.

At the 50-word cutoff,

$$26.75-17.82=\mathbf{8.93}$$

BLEU points separate them.

So the result is not merely that a larger training-length cutoff helped. Within each matched cutoff, RNNsearch substantially outperformed RNNencdec. RNNsearch-30 even scored higher than RNNencdec-50 despite being trained only on sentences of up to 30 words.

That is strong experimental evidence that the architectural change mattered.

But it should not be read as a perfectly isolated experiment in which only an attention module was switched on. RNNsearch also replaces the baseline encoder with a bidirectional recurrent encoder, so the experiment compares complete architectures rather than attention alone.

28.45 and 36.15 answer different questions

The paper reports two BLEU columns, and they should not be mixed.

The longer-trained model, RNNsearch-50*, achieved:

$$\boxed{28.45}$$

BLEU on the full WMT14 test set, containing 3,003 sentences.

It achieved:

$$\boxed{36.15}$$

on a restricted subset containing only sentences for which neither the source nor the reference translation contained an unknown vocabulary item. During this evaluation, the neural systems were also prevented from generating the UNK token.

The complete comparison was:

These columns answer different questions.

28.45 asks:

How well does the model translate the complete test set, including sentences containing words outside its shortlist?

36.15 asks something closer to:

How well does it translate when the evaluation avoids this particular unknown-word problem?

This distinction mattered because the neural systems used vocabularies containing only the 30,000 most frequent words in each language. Everything outside those shortlists became UNK.

The phrase-based Moses system scored 33.30 on the complete set and 35.63 on the no-UNK subset.

RNNsearch-50* therefore slightly exceeded Moses on that restricted subset:

$$36.15 > 35.63$$

but Moses remained substantially ahead on the full test set:

$$33.30 > 28.45.$$

There is another important qualification: Moses used an additional 418-million-word monolingual corpus, whereas the neural models were trained from the selected parallel data without that extra monolingual corpus.

The comparison was therefore impressive, but 28.45 and 36.15 should never be presented as interchangeable headline scores.

The length experiment attacks the bottleneck hypothesis

Recall the original conjecture:

$$\text{long sentence} \rightarrow \boxed{\text{one fixed vector}} \rightarrow \text{decoder}$$

may become increasingly difficult as the source grows.

The length-binned results provide evidence consistent with that hypothesis.

The fixed-context RNNencdec deteriorated sharply as sentence length increased. In contrast, RNNsearch was considerably more robust, and the authors reported that RNNsearch-50 showed no comparable deterioration even for sentences of 50 words or more.

Conceptually:

Translation
quality
   ▲
   │
   │  RNNsearch ─────────────────╲
   │
   │  RNNencdec ───────╲
   │                    ╲
   │                     ╲____
   │
   └──────────────────────────────►
              sentence length

The authors' explanation is intuitive.

RNNencdec has to preserve the information needed for the entire sentence in one representation:

$$c.$$

RNNsearch can instead leave information distributed across:

$$h_1,h_2,\ldots,h_{T_x}$$

and retrieve different portions when they are useful.

The paper therefore argues that RNNsearch does not need to encode a long sentence perfectly into one vector; it can encode local regions well and retrieve them as decoding proceeds.

But this result supports the explanation rather than proving a single cause.

The experimental change is approximately:

$$\text{fixed-context unidirectional encoder-decoder}$$

versus

$$\text{bidirectional encoder} + \text{dynamic attention} + \text{decoder}.$$

Because several architectural properties change together, Figure 2 does not by itself establish that every improvement comes solely from removing the fixed context vector.

It nevertheless provides strong evidence that the RNNsearch architecture as a whole handles increasing source length much better.

The heatmap is evidence, not an oracle

One of the most influential features of the paper is that its internal weighting mechanism can be visualized.

For each generated target word (y_i), the model has attention weights

$$\alpha_{i1},\alpha_{i2},\ldots,\alpha_{iT_x}.$$

Putting those values into a matrix gives a source-target heatmap:

                    SOURCE
               x₁   x₂   x₃   x₄   x₅

          y₁   ██   ░    ░    ░    ░
          y₂   ░    ██   ▒    ░    ░
TARGET    y₃   ░    ▒    ██   ░    ░
          y₄   ░    ░    ░    ▒    ██

Each cell is an attention weight:

$$\alpha_{ij}.$$

A bright cell means that the annotation around source position (j) received a large weight while the model was producing target position (i). Figure 3 of the paper shows mostly monotonic English-to-French correspondences together with interesting non-monotonic patterns caused by reordering.

For example, the paper discusses the translation of:

European Economic Area

into the reordered French phrase:

zone économique européenne

The heatmap shows the model moving between relevant source positions rather than being constrained to a simple left-to-right alignment.

The famous article example is even more revealing.

To translate:

“the man”

into:

“l'homme”

the correct French article depends not merely on the English word the, but also on information associated with the following noun.

The soft alignment can therefore place weight on more than one annotation:

       the        man
        │          │
        └────┬─────┘
             ▼
           "l'"

That is exactly the kind of decision that a one-to-one hard word alignment describes poorly.

But the heatmap should not be treated as an oracle explaining the model's reasoning.

A large value of

$$\alpha_{ij}$$

tells us that annotation (h_j) received a large coefficient in constructing (c_i).

It does not, by itself, prove:

$$\text{“target word }i\text{ was caused by source word }j\text{.”}$$

Remember that (h_j) itself is contextual:

$$h_j= [\overrightarrow{h}_j;\overleftarrow{h}_j].$$

It already contains information gathered from surrounding source words.

So even a sharply concentrated attention weight is a weight on a contextual representation, not a clean pointer to an isolated word.

Later work reinforced this caution. Analyses of neural machine translation found that encoder-decoder attention and traditional word alignment are not always the same thing, and some models' attention weights can be poor direct word-alignment estimators.

A better interpretation is:

$$ \boxed{ \text{attention heatmap}

\text{inspectable evidence about information routing} } $$

rather than:

$$ \boxed{ \text{attention heatmap}

\text{complete causal explanation} } $$

Attention also introduces an exhaustive search cost

There is another tradeoff hidden inside the elegant heatmap.

For every target position (i), the decoder computes an alignment score against every source annotation:

$$e_{ij}=a(s_{i-1},h_j).$$

If there are (T_x) source positions and (T_y) target positions, the alignment mechanism evaluates approximately

$$T_xT_y$$

source-target pairs.

For example:

$$30\times30=900$$

alignment comparisons,

while:

$$100\times100=10,000.$$

The authors explicitly called this a drawback. They considered it manageable for machine translation because typical input and output sentences were around 15–40 words, but noted that exhaustive scoring could restrict the method on other tasks.

So Bahdanau attention removes one restriction:

$$\text{everything must pass through one }c$$

by accepting another:

$$\text{compare each output step with every source position}.$$

For machine translation in 2014, that was a very productive trade.

The paper therefore leaves us with three distinct pieces of evidence:

$$\boxed{\text{better BLEU}}$$

$$\boxed{\text{greater robustness to sentence length}}$$

$$\boxed{\text{inspectable, linguistically plausible soft alignments}}$$

None alone proves that a single fixed vector was the cause of every failure.

Together, however, they provided a compelling case that letting the decoder dynamically retrieve source information was substantially more effective than forcing the entire source sentence through one reusable context vector.