Lightweight Adaptation Techniques for Multimodal Large Models

Pre‑trained language models such as BERT and GPT‑3 have achieved excellent results in NLP, and multimodal pre‑trained models like ViLBERT, CLIP and OFA have demonstrated strong performance on downstream tasks. However, applying open‑source multimodal large models in industry faces three main challenges: language mismatch, low domain compatibility, and differing optimization objectives.

Language adaptation : A lightweight adapter‑based pipeline replaces the English text encoder with a Chinese encoder while keeping the visual encoder frozen. The adapter maps Chinese BERT outputs into the English encoder space using 1‑2 Transformer layers and two linear layers, drastically reducing training cost and preserving accuracy.

Experimental results show that zero‑shot CLIP with a small amount of high‑quality translated data reaches 56% accuracy, while raw Chinese data yields only 18%. After applying the Chinese adapter, CLIP recovers to 55% accuracy. On the COCO‑CN Text2Img benchmark, the adapted CLIP achieves an mR of 73.5, surpassing the dedicated Chinese model Wukong (mR 72.2) and matching the performance of large‑scale multilingual models.

Domain adaptation : Building on the language adapter, a hard‑sampling adapter introduces instance weighting to focus training on difficult samples. Using only 10 k domain‑specific pairs, the hard‑sampling adapter outperforms both a 6 M‑sample end‑to‑end pretrained model and a continual‑learning baseline, achieving higher accuracy in cosmetics and food domains.

Optimization‑goal adaptation : Structured prompts are introduced to model General, Domain, and Instance information separately, combined with a visual‑guided attention mechanism for learnable instance prompts. Few‑shot image classification experiments demonstrate consistent gains over baseline methods, achieving state‑of‑the‑art performance with only 12.5% of the data needed by a ResNet model.

Q&A highlights : The adapter uses a small number of parameters (1‑2 Transformer layers). Domain information refers to category labels (e.g., "butterfly"), while instance information captures visual attributes (e.g., "on a flower"). The approach can be transferred to other fields such as life sciences if the underlying multimodal knowledge is sufficiently generic.