Tencent Cloud Developer
Mar 3, 2022 · Artificial Intelligence
Model Distillation for Query-Document Matching: Techniques and Optimizations
We applied knowledge distillation to a video query‑document BERT matcher, compressing the 12‑layer teacher into production‑ready 1‑layer ALBERT and tiny TextCNN students using combined soft, hard, and relevance losses plus AutoML‑tuned hyper‑parameters, achieving sub‑5 ms latency and up to 2.4% AUC improvement over the original model.
ALBERTAutoMLBERT
0 likes · 12 min read