Integrating Retrieval and Generation Tasks for Deep Semantic Matching in Xianyu Search

Deep semantic matching is crucial for relevance calculation in Xianyu search. This article presents a recent attempt that jointly trains retrieval and generation tasks to improve matching performance.

The approach builds on the SimBERT model, adapting it with a BERT backbone to create a model called SimBert.

In production, Xianyu uses two strategies: (1) a dual‑tower BERT model with contrastive loss and hard‑negative sampling, and (2) a later‑fusion architecture that adds multi‑layer fully‑connected networks on top of the dual‑tower outputs and employs a teacher BERT model for distillation.

SimBert inherits the later‑fusion structure but adds an auxiliary generation task in the item tower, as illustrated in the accompanying diagram.

The generation task masks the CLS token of the item tower to prevent feature leakage and enforces an autoregressive mask so each generated token can only attend to previous tokens. This is achieved by reshaping BERT’s attention‑mask matrix.

Training proceeds in two stages. During pre‑training, a multi‑task loss combines InfoNCE contrastive loss for the matching task with the generation loss. Fine‑tuning uses high‑quality human‑labeled data, adds the later‑fusion fully‑connected layers, and optimizes a binary classification loss together with the generation loss.

Data construction mirrors the training stages: pre‑training uses click‑log query‑title pairs with hard negative sampling, while fine‑tuning relies on manually annotated {Query, Title, Label} triples. The generation task shares the same positive pairs.

Offline metrics and online A/B tests show a 3.6% increase in relevance accuracy, with a 3.88% reduction in bad‑case rate for top queries and a 6.15% reduction for random queries.

Future work includes expanding and cleaning training data, incorporating knowledge and keyword supervision, and adding multi‑modal signals such as item descriptions and structured attributes.

References: 1) Xianyu search relevance article; 2) SimBERT paper; 3) UniLM (arXiv:1905.03197).