Project 3: Attention and Transformers

Experimental Setup

Dataset. The English–French translation corpus from d2l (~180 sentence pairs, max 9 tokens per sentence). No separate validation split is used; training loss is monitored directly. Evaluation. Sentence-level BLEU (k=2 n-grams) averaged over four held-out test pairs from Project 2, using greedy decoding. Because the test set is very small, BLEU scores should be interpreted as rough indicators rather than precise performance measures. Optimiser. Adam with gradient clipping (max norm = 1.0). All experiments are single-run (no seed averaging), so minor score differences may reflect randomness.

Discussion

Hidden size is the primary differentiator. Configs A and C share the same learning rate and layer count but differ in hidden dimension (256 vs 512). The larger hidden size in C pushes BLEU from 0.75 to 0.91, suggesting that on this small dataset, the model needs sufficient capacity to memorise the training patterns that matter for correct translation.

Loss and BLEU can be decoupled. An interesting finding is that lower training loss does not guarantee higher BLEU. Config B achieves the lowest final loss (0.213) yet the same BLEU (0.75) as Config A (loss 0.228). Conversely, the best-BLEU Config C has a slightly higher loss (0.230) than both A and B. This suggests that token-level cross-entropy and sequence-level BLEU optimise different aspects of translation quality—a model can fit the token distribution well but still make critical errors on specific words (e.g., “he's calm”), while a model with slightly worse average loss may translate the key content words correctly.

Learning rate is critical for convergence within a fixed epoch budget. Config D (lr = 0.001) barely reaches useful translations by epoch 50—its loss is still 0.82, roughly 4× higher than other configs. The small dataset does not benefit from cautious optimisation; lr = 0.005 converges comfortably.

Dropout has limited effect on this corpus. Configs A (dropout 0.2) and C (dropout 0.3) reach nearly identical final losses despite different regularisation strengths. With only ~180 sentence pairs, overfitting is not the bottleneck—representational capacity is.

Discussion

Short sentences show near-diagonal alignment. For “go .” → “va !”, the heatmap is almost perfectly diagonal: “va” attends to “go” and “!” attends to “.”. This confirms the model learns a sensible word-level alignment for simple translations.

Reordering is visible in longer sentences. In “i'm home .” → “je suis chez moi .”, the model attends to “home” for both “chez” and “moi”, reflecting the French prepositional phrase structure. The attention spreads across multiple source positions rather than locking onto a single word, showing the model captures phrase-level rather than word-level correspondence.

The <eos> token acts as a generation signal. In several heatmaps the final decoding steps attend heavily to the source <eos> marker. This is a learned heuristic: the model uses the boundary token to trigger its own stopping decision.

Discussion

Multi-head attention achieves comparable BLEU at a smaller hidden size. The 2-head configuration (hidden=256) matches the Bahdanau model that required hidden=512 to reach the same BLEU. While this hints at better parameter utilisation by the dot-product mechanism, we did not control for total parameter count, so the observation should be interpreted cautiously. The dot-product formulation is computationally cheaper than additive scoring, which may become relevant at larger scale.

More heads do not always help on small data. On this tiny dataset (~180 sentence pairs), increasing from 2 to 4 or 8 heads yields no BLEU improvement. Each head operates in a subspace of dimension d_model/h; with 8 heads and hidden=256, each head only has 32 dimensions, which may be too narrow to capture meaningful patterns from such limited training data. The loss curves converge to similar values regardless of head count, confirming that the bottleneck is data size, not model capacity.

Dot-product vs. additive attention. Both attention types achieve comparable results here. The theoretical advantage of scaled dot-product (parallelisable matrix multiplications) becomes significant only at scale. On this small benchmark the two mechanisms are functionally equivalent in quality.

Discussion

Partial specialisation is visible. Across the three sentences, some heads consistently show sharper, more localised attention (attending to 1–2 source positions), while others spread attention more broadly. For “i lost .”, one head focuses almost exclusively on “lost” when generating “perdu”, while another distributes weight across “i” and the punctuation.

Redundancy is common on small datasets. Several head pairs show very similar patterns, particularly for the short sentence “go .”. With only two meaningful source tokens, there is not enough signal to drive four heads toward distinct roles. This redundancy explains why reducing to 2 heads actually matches or exceeds 4-head performance on this benchmark.

Head diversity increases with sentence length. The longest sentence (“he's calm .”) shows the most inter-head variation. This aligns with the intuition that multi-head attention shines when the model needs to simultaneously track multiple relationships (e.g., subject–verb agreement and punctuation mapping).

Discussion

All three models converge to comparable BLEU on this dataset. The English–French corpus has only ~180 sentence pairs with short sequences (max 9 tokens). At this scale, the representational advantages of the Transformer do not have room to manifest; even a simple GRU with Bahdanau attention can memorise the training set. The Transformer's loss curve is smoother and converges slightly slower in early epochs (due to the lower learning rate of 0.001 needed for stability), but reaches the same final level.

The Transformer would likely outperform on larger, longer-sequence data. Its key advantages—parallel computation, global attention over all positions in a single layer, and explicit positional encoding—become critical when sentences are long and the vocabulary is large. On this toy benchmark, all models are effectively equivalent.

RNN-based models train faster in wall-clock time here. With only 9 time steps and tiny batches, the GRU models finish training in seconds. The Transformer, while theoretically parallelisable, has higher per-step overhead from multi-head attention and layer normalisation on such small tensors.

Discussion

Positional encoding has minimal impact on these short sequences. Removing sinusoidal positional encodings barely changes BLEU or the loss trajectory. Strictly speaking, self-attention without positional encoding is permutation-invariant and cannot distinguish word order from architecture alone. However, on sequences of at most 9 tokens, order can be partially inferred from content cues (e.g., <bos> always starts the sequence, punctuation appears at the end), and the dataset is small enough that the model memorises the training pairs regardless. On longer sequences where word order is ambiguous, positional encoding would become essential.

Removing decoder self-attention does not degrade BLEU on this dataset. The “no decoder self-attention” variant achieves BLEU 1.00 on the four test sentences. However, this result should be interpreted with caution: the evaluation set consists of only four short sentence pairs, and this is a single-run experiment without seed averaging, so the difference from the full model (0.91 vs 1.00) likely reflects random variation rather than a genuine architectural advantage. Mechanistically, without decoder self-attention, the model becomes a source-conditioned model where each target position depends only on the encoder output and the FFN, without explicit modelling of target-side dependencies. On formulaic, short target sentences, this can work well—but on longer sequences requiring coreference resolution or long-distance agreement, decoder self-attention would be critical.

Loss and BLEU again diverge. The “no decoder self-attention” variant has the highest final loss (0.267) among the three configurations, yet the highest BLEU. This mirrors the pattern observed in Part 1 and reinforces that token-level loss and sequence-level BLEU measure different aspects of translation quality. A model with fewer parameters may have higher average loss but still produce correct translations for the specific test sentences.

Takeaway: component importance is data-regime-dependent. Both positional encoding and decoder self-attention are fundamental to the Transformer's success on large-scale tasks (e.g., WMT benchmarks, language modelling), but on a toy dataset with short sequences they are redundant. This ablation illustrates why architectural conclusions must be drawn at the target task's scale—results on ~180 sentence pairs do not generalise to production-scale translation.