Understanding Transformers: From NLP Challenges to Architecture and Core Mechanisms

1. The Rise of Artificial Intelligence – In 1950 Alan Turing proposed the imitation game (Turing Test) that sparked the idea of machines exhibiting intelligent behavior.

2. NLP Development – Understanding human language is the first step for intelligent machines; early NLP relied on rule‑based models that required extensive manual effort and could not handle unseen queries.

Rule‑Based Models – Effective in narrow domains but suffer from scalability and conflict issues.

Statistical Models – Based on the Markov assumption, n‑gram models capture word probabilities but face the long‑distance dependency problem as n grows.

Neural Network Models – CNN and RNN introduced learning from data; RNNs alleviate some long‑range issues but encounter gradient vanishing/explosion.

LSTM – Adds memory cells and gates (input, output, forget) to mitigate gradient problems and capture longer contexts.

3. Transformer

The Transformer, introduced by Google in the 2017 paper "Attention Is All You Need," replaces recurrence with attention mechanisms, enabling parallel training.

Word Embedding

Maps words to high‑dimensional vectors (e.g., Word2Vec, GloVe) so that semantically similar words are close in vector space.

Position Embedding

Since Transformers lack inherent order, sinusoidal position embeddings are added to encode token positions.

Self‑Attention Mechanism

Four steps:

Generate Q (query), K (key), V (value) vectors for each token.

Compute scaled dot‑product of Q with Kᵀ to obtain attention scores.

Apply softmax to normalize scores into weights.

Weight V by these scores to produce the output.

Multi‑Head Attention

Multiple self‑attention heads run in parallel, each learning different relational aspects; their outputs are concatenated and linearly projected.

Add & Norm Layer

Residual connections preserve original information, while layer normalization stabilizes training.

Feed‑Forward Layer

Applies a position‑wise fully connected network with non‑linear activation to further transform features.

Encoder & Decoder

Encoder blocks consist of Multi‑Head Attention + Add & Norm + Feed‑Forward; Decoder blocks add a masked Multi‑Head Attention to prevent future token leakage and another attention over encoder outputs.

Final softmax layer predicts the next token.

Transformer Summary

Enables parallel training unlike RNNs.

Requires position embeddings to retain order information.

Core is self‑attention using Q, K, V matrices.

Multi‑Head Attention captures diverse relationships between words.

Add & Norm and Feed‑Forward layers improve stability and capacity.

References:

GitHub – Learn NLP with Transformers

Tech Dewu Article

Zhihu Post

Attention Is All You Need (arXiv)