Explainable LLM-driven Multi-dimensional Distillation for E-Commerce Relevance Learning

In the era of large‑model driven transformation across internet industries, this work investigates whether search advertising can leverage large language models (LLMs) to continuously increase revenue and improve consumer shopping experience. The authors present a novel framework for e‑commerce relevance modeling that combines an explainable LLM (ELLM‑rele) with a multi‑dimensional knowledge distillation (MKD) pipeline.

The proposed framework addresses two major challenges of LLM‑based relevance modeling: (1) the massive parameter size and computational cost that hinder direct online deployment, and (2) the black‑box nature of LLM reasoning, which makes it difficult to extract and reuse the rich knowledge embedded in the model. To overcome these issues, the authors decompose relevance judgment into a series of fine‑grained sub‑tasks (e.g., category match, brand match, model match) and reconstruct the task as a Chain‑of‑Thought (CoT) reasoning process.

ELLM‑rele is trained by first generating high‑quality CoT annotations using a collaborative labeling strategy that combines two powerful LLMs (Qwen2‑72B and LLaMA3‑70B). The annotated CoT data are then used to fine‑tune a lightweight 7B LLM, enabling it to produce both relevance scores and interpretable reasoning steps while meeting latency requirements for online serving.

The MKD component transfers the knowledge of ELLM‑rele to student models deployed online. Two complementary distillation streams are employed: (a) relevance‑score distribution distillation, which converts the teacher’s token‑level probability distribution into a continuous relevance score for the student; and (b) CoT knowledge distillation, which parses the reasoning chain into token‑level supervision (BIO tags for interactive models or attention‑adjustment factors for representation‑based models). Both streams are jointly optimized with the original cross‑entropy loss.

Extensive offline experiments on a manually labeled relevance test set show that ELLM‑rele outperforms both black‑box LLMs and traditional representation‑ or interaction‑based models. The MKD framework further boosts the performance of various student models, demonstrating strong generalization across modeling paradigms. Online A/B tests confirm significant improvements in click‑through rate, conversion rate, and long‑tail sample performance, leading to a full rollout in Taobao’s search advertising system.

The paper, titled “Explainable LLM‑driven Multi‑dimensional Distillation for E‑Commerce Relevance Learning,” has been accepted at WWW’25. The authors provide detailed implementation details and release the code and data for reproducibility.