Multimodal Pre‑training Techniques and Applications – Overview, OPPOVL Dataset, Architecture, and Performance

Multimodal pre‑training aims to achieve more human‑like interaction by jointly learning from visual and textual modalities, addressing the growing demand for cross‑modal AI services. Traditional single‑modal pre‑training (e.g., BERT, ResNet) is extended to bridge semantic gaps between images and text.

The OPPOVL dataset was built by collecting over 30 million Chinese image‑text pairs from news, encyclopedias, and web sources. Data cleaning involved image size filtering, removal of pornographic or politically sensitive content, JPEG compression, and text filtering (non‑Chinese removal, sensitive‑word masking, personal‑name anonymization, and elimination of boilerplate strings). Low‑correlation pairs were discarded using a BriVL similarity score threshold of 0.3, and duplicate pairs were removed while preserving distinct images for identical captions.

OPPOVL adopts a dual‑stream architecture: a custom visual backbone (CETNet) for image encoding and BERT‑Base for text encoding. The two encoders are jointly trained with a bidirectional image‑text contrastive loss, optionally combined with single‑modal self‑supervised tasks (SSL, MLM). This design simplifies adding new pre‑training tasks and yields better performance than comparable single‑stream models at similar scale.

Training optimizations include mixed‑precision arithmetic, gradient accumulation, and high‑throughput data loading via WebDataset. Additional synthetic captions generated by large‑scale generative models augment the original data, improving data utilization. Momentum distillation is employed to refine contrastive learning, though it incurs extra compute and is recommended for smaller models or fine‑tuning.

Experimental results show that OPPOVL outperforms CLIP‑Res50 on CC3M, CC12M, and YFCC15M benchmarks, achieving higher retrieval scores with only a fraction of the training data. On a 40 million‑sample Chinese dataset, OPPOVL surpasses Huawei’s WuKong model despite using less than half the data, demonstrating the effectiveness of high‑quality data and architecture.

Potential applications span photo classification, captioning, multimodal search, virtual content creation for OPPO’s metaverse, and other cross‑modal services. Model variants of different sizes enable easy adaptation to diverse downstream tasks with minimal fine‑tuning.

The Q&A section addresses common concerns such as data generality, domain‑specific fine‑tuning data requirements, cleaning consistency, inference speed, and the role of large‑scale pre‑training in downstream performance.

In summary, multimodal pre‑training is a promising direction for next‑generation intelligent agents, with key factors including model architecture, pre‑training task design, high‑quality data pipelines, and efficient training strategies.