Data Management in Large Language Model Training: Overview, Pre‑training, SFT, and Future Challenges

The presentation is organized into four main chapters: (1) an overview of data management in large language model (LLM) training, (2) data management during the pre‑training phase, (3) data management in the supervised fine‑tuning (SFT) phase, and (4) challenges and future directions.

Pre‑training data management focuses on domain composition (web, Wikipedia, code, dialog, academic, math, etc.), data quantity (scaling laws by Kaplan and Chinchilla), and data quality (filtering, deduplication, harmful‑information removal, diversity, and data age). Methods such as DSIR, DoReMi, DoGE, and Data‑Mixing Laws are discussed for optimizing domain ratios, while N‑gram hashing, neural‑network‑based deduplication, and semantic clustering are presented as deduplication techniques.

SFT data management emphasizes multi‑task instruction fine‑tuning, instruction quality (classifier‑based, rule‑based, metric‑based), diversity (tag‑based metrics, Rouge‑L, embedding distance), and complexity (label count, tree‑instruct, Evol‑instruct). Dynamic data learning strategies such as early stopping, data pruning, task‑wise active search, and curriculum learning (easy‑to‑hard ordering) are highlighted.

Future challenges include building a universal data‑management framework, bias and harmful content mitigation, multimodal instruction data handling, self‑exploratory data management for large‑scale interaction data, efficient synthetic‑data filtering, fine‑grained data pipeline design, and conflict data isolation.

References: a recent survey (arXiv:2312.01700) and an accompanying GitHub repository (github.com/ZigeW/data_management_LLM).