Enhancing Taobao Display Advertising with Multimodal Representations: Challenges, Approaches, and Insights

The paper addresses the limitation of traditional ID‑based recommendation models, which cannot capture item semantic information, by exploring how native multimodal content (images and text) can be incorporated to improve click‑through‑rate (CTR) prediction for Taobao display advertising.

Background: Existing large‑scale sparse ID features combined with MLPs dominate current estimation models, but they fail to model semantic similarity between items. The authors identify two key questions: how multimodal information can boost model performance and how to release its benefits within an ID‑centric framework.

SCL – Semantic‑aware Contrastive Learning: A pre‑training task that pulls together multimodal representations of semantically similar item pairs (derived from user search‑to‑purchase behavior) and pushes apart unrelated pairs. Positive pairs are constructed as <user search image, purchased item image> or <user search text, purchased item title> . Negative samples are drawn from a MoCo memory bank, and InfoNCE loss is applied.

SimTier: To simplify multimodal usage, SimTier discretizes the similarity scores between a target item and each historical behavior item into N bins, counts occurrences per bin, and forms an N‑dimensional vector representing the similarity distribution. This vector is concatenated with ID embeddings and fed to downstream MLPs.

MAKE – Multimodal Knowledge Extraction: MAKE decouples the optimization of multimodal parameters from ID‑based parameters by first pre‑training a multimodal‑only CTR model for several epochs, then injecting the learned multimodal knowledge into the full model. This resolves the epoch‑mismatch between ID and multimodal branches.

Experiments: The authors compare SCL with CLIP‑O, CLIP‑E, and other baselines, showing superior accuracy and semantic discrimination. They also evaluate SimTier, MAKE, and their combination against ID‑only, raw vector, and similarity‑based methods, reporting up to +1.25 % GAUC and +0.75 % AUC improvements, especially for long‑tail items.

Online Deployment: A real‑time multimodal encoder service generates embeddings for newly created items within seconds, achieving >99 % feature coverage and reducing cold‑start latency. Since mid‑2023, multimodal features have been fully deployed in Alibaba’s ad ranking pipeline, delivering +3.5 % CTR, +1.5 % RPM, and +2.9 % ROI gains, with even larger lifts for fresh ads.

Conclusion & Outlook: Multimodal representations effectively complement ID features, delivering substantial business impact. Future work includes integrating multimodal signals with long‑sequence modeling, large‑model world knowledge, and generative recommendation techniques.