Cross‑Modal Image‑Text Representation: The Zero Dataset and R2D2 Pre‑training Framework

In the Internet era, text, images and videos are tightly coupled, making image‑text cross‑modal representation a fundamental problem for tasks such as text‑to‑image retrieval, video recommendation and article illustration.

The classic CLIP model uses a dual‑tower architecture where separate image and text encoders map inputs to a shared embedding space; its success is largely due to billions of image‑text pairs. Chinese improvements such as WenLan (1.0/2.0) and Huawei's Wukong add fine‑grained token alignment and larger multilingual data.

Two main design philosophies exist: (1) dual‑tower models that keep image and text encoders separate for fast retrieval, and (2) single‑tower models that fuse modalities early for richer interaction, at the cost of higher computation.

Zero Dataset : Built from 360's search click logs, the pipeline filters billions of candidates down to 250 million high‑quality pairs, then randomly samples a 23 million and a 2.3 million subset for pre‑training. Zero also provides five downstream benchmarks (long/short text‑image matching, long/short text‑image retrieval, and a refined Chinese Flickr30K‑CNA). The dataset includes query, image, title, surrounding text and URL, and is publicly released.

R2D2 Framework : R2D2 combines a CLIP‑style dual‑tower (ViT image encoder, RoBERTa text encoder) with a cross‑attention single‑tower for fine‑grained interaction. Four losses are jointly optimized: (1) global contrastive loss (GCPR), (2) fine‑grained image‑to‑text classification loss, (3) fine‑grained text‑to‑image classification loss, and (4) masked language modeling (MLM) loss. Additionally, target‑guided and feature‑guided distillation incorporate soft teacher signals and a historical negative‑sample queue. The training proceeds in two stages: a fast pre‑ranking dual‑tower pass followed by a more expensive single‑tower re‑ranking.

Experimental Results : On the five Zero downstream tasks, models trained on 2.3 M data already surpass previous SOTA, 23 M data achieve full superiority, and the 250 M pre‑training set yields further gains (e.g., +4.7% on Flickr30K‑CN, +5.4% on COCO, +6.3% on MUGE). Ablation studies show that removing the joint MLM‑fine‑grained loss or the distillation components degrades performance, confirming their importance.

Business Deployment : The dual‑tower part of R2D2 is used for large‑scale image‑text retrieval in 360 Search, while the single‑tower re‑ranking improves precision in ad placement, recommendation and video understanding. The models and code are open‑source, encouraging further industry adoption.

Conclusion : Zero provides a fair, high‑quality benchmark for Chinese image‑text research, and R2D2 demonstrates how combining dual‑tower efficiency with single‑tower interaction and multi‑task distillation yields state‑of‑the‑art performance. Both the dataset and the framework are released for the community.

Finally, the speaker announces the launch of Carbon Silicon AI, a startup aiming to fuse cutting‑edge AI with life‑science research, and invites collaborators.