Comprehensive Survey of Pre-trained Models for Natural Language Processing

Pre‑trained models (PTMs) have revolutionized NLP by leveraging large amounts of unlabeled data to learn general language representations that can be fine‑tuned for downstream tasks. The article begins with a motivation for pre‑training, highlighting benefits such as richer language understanding, better initialization, and regularization.

PTMs are divided into two major paradigms: shallow (non‑contextual) word embeddings like word2vec, GloVe, and NNLM, and deep contextual encoders such as ELMo, GPT, BERT, XLNet, etc. Shallow embeddings are static and suffer from OOV and polysemy issues, while contextual encoders produce token representations that depend on surrounding context.

Within contextual encoders, three families of language‑modeling objectives are described: (1) Autoregressive language models (LM) that predict the next token sequentially, (2) Denoising auto‑encoders (DAE) that mask tokens and predict them bidirectionally (e.g., BERT, RoBERTa), and (3) Permuted language models (PLM) that randomize the factorization order (e.g., XLNet). Additional contrastive‑based objectives such as Deep InfoMax, Replaced Token Detection, and Sentence Order Prediction are also covered.

The survey outlines several extensions of PTMs: incorporation of external knowledge (ERNIE‑THU, LIBERT, SenseBERT, etc.), model compression techniques (pruning, quantization, parameter sharing, knowledge distillation), multimodal pre‑training (VideoBERT, ViL‑BERT, etc.), domain‑specific pre‑training (BioBERT, SciBERT, Clinical‑BERT), and multilingual or language‑specific models (mBERT, XLM, CamemBERT, etc.).

For transfer learning, the article discusses strategies such as selecting appropriate pre‑training tasks, model architectures, and data, choosing which layers to transfer (embedding, top‑layer, or all layers), and deciding between feature extraction and fine‑tuning. Advanced fine‑tuning methods include multi‑stage training, multi‑task learning, adapter modules, and layer‑wise freezing.

Finally, the article identifies open challenges: scaling PTMs further, task‑oriented pre‑training and compression, improving transformer efficiency for longer sequences, enhancing knowledge transfer during fine‑tuning, and increasing interpretability and reliability of PTMs.