Understanding BERT: From Encoder-Decoder to Transformer and Attention

1. Introduction

In 2018 Google AI released BERT (Bidirectional Encoder Representations from Transformers), achieving state-of-the-art results on 11 NLP tasks and surpassing human performance on the SQuAD reading-comprehension benchmark.

The article introduces BERT’s architecture and its underlying principles. Before diving into BERT, it reviews the general Encoder-Decoder framework, the attention mechanism, and the Transformer encoder that BERT builds upon.

2. Encoder-Decoder Framework

Encoder-Decoder is a common seq2seq architecture used in speech recognition, machine translation, dialogue systems, image captioning, etc. The encoder converts an input sequence into a vector representation, and the decoder generates an output sequence from that vector.

Traditional encoders use RNNs, which can lose information from the beginning of long inputs. This motivates the attention mechanism.

3. Attention Mechanism

Attention assigns different weights to different parts of the input, focusing on relevant information. The process involves three elements: query, key, and value.

Steps: compute similarity between query and key, apply softmax to obtain attention weights, and use the weights to compute a weighted sum of the values.

Compute similarity between query and key.

Softmax‑normalize the similarities to get attention distribution.

Multiply the distribution with the values and sum to obtain the attention vector.

Various similarity functions exist (dot‑product, scaled dot‑product, etc.).

3.2 Self-Attention

Self-attention replaces RNN/CNN in Transformers. For each token, linear projections produce query, key, and value vectors (q_i = W_q·x_i, k_i = W_k·x_i, v_i = W_v·x_i). Scores are computed as q_i·k_j, normalized with softmax (often scaled by √d_k), and used to weight the values:

z_i = Σ softmax(q_i·k_j) * v_j

Multi-head attention runs several self-attention heads in parallel and concatenates their results.

4. Transformer

The Transformer consists of an encoder stack and a decoder stack. Each encoder layer contains a multi-head attention sub-layer followed by a feed-forward network; the decoder adds a masked self-attention sub-layer and an encoder-decoder attention sub-layer.

Residual connections (Add) and Layer Normalization are applied after each sub-layer.

Positional embeddings are added to token embeddings to inject sequence order information.

5. BERT

BERT uses only the Transformer encoder, stacking N encoder blocks (12 in BERT‑base, 24 in BERT‑large). Input tokens are summed with segment and positional embeddings. Special tokens [CLS] and [SEP] mark sentence boundaries.

Training consists of two stages:

Pre‑training with masked language modeling (15 % of tokens are masked; 80 % replaced by [MASK], 10 % by a random token, 10 % left unchanged).

Fine‑tuning on downstream tasks such as text classification, question answering, or natural‑language inference, typically using the [CLS] representation as input to a classifier.