Parameter-Efficient Sparsity Training for the PLUG Large-Scale Language Model

In March 2021, Alibaba DAMO Academy released PLUG, a 27‑billion‑parameter Chinese language understanding and generation model. Deploying such massive models is challenging due to memory consumption and inference latency.

The authors propose a sparse training solution, Parameter‑Efficient Sparsity Training (PST) , which includes two key components:

Sparse Tuning : learns a sparse sub‑network from a small amount of data, achieving fine‑tuning quality comparable to the full model.

Sparse Serving : builds an efficient inference engine that leverages sparse matrix operations instead of dense kernels.

Related work on model compression is reviewed, covering weight sharing, knowledge distillation, low‑rank factorization, pruning, and quantization. Evaluation metrics such as FLOPs, inference time, speed‑up ratio, parameter count, model size, carbon footprint, fidelity, and robustness are discussed.

Experiments on NLU (GLUE/CLUE) and NLG benchmarks demonstrate that PST can prune up to 99% of parameters while maintaining or even improving accuracy compared with magnitude pruning, movement pruning, and L0 regularization baselines. Detailed analyses show smoother weight and score distributions and favorable sparsity patterns.

The paper concludes that PST enables practical deployment of ultra‑large language models like PLUG, achieving high sparsity with minimal performance loss, and outlines future directions including faster sparsity algorithms, integration with distillation and quantization, and hardware‑aware acceleration.

A Q&A section addresses practical concerns such as model export, hardware support for sparsity, and the readiness of PST on Alibaba’s AliceMind platform.