SPACE and Proton: Semi‑Supervised Knowledge Injection and Probing‑Tuning for Pretrained Conversational AI Models

The Alibaba DAMO‑ConvAI team, a pioneer in intelligent dialogue since 2014, presents their research on large‑scale conversational AI, focusing on knowledge injection and utilization within pretrained dialogue models.

First, the article contrasts pretrained language models with pretrained dialogue models, highlighting that dialogue data is more conversational, multi‑turn, knowledge‑constrained, and strategy‑driven, requiring specialized modeling beyond generic language pretraining.

It then surveys the progress of pretrained dialogue models, covering understanding (e.g., ConveRT, TOD‑BERT, DialogBERT) and generation (e.g., DialoGPT, Meena, Blender, PLATO‑2), and notes the significant performance gains these models bring to dialogue tasks.

To address the need for knowledge in dialogue, the authors propose a two‑step approach: (1) injecting knowledge into pretrained dialogue models via a semi‑supervised pretraining framework called SPACE, and (2) explicitly extracting and leveraging the learned knowledge for downstream tasks through a probing‑tuning method named Proton.

SPACE treats annotated knowledge as a valuable resource and combines a small amount of labeled data with massive unlabeled data using discriminative, generative, and contrastive learning. The contrastive loss aligns representations of the same sample under different dropout masks, enabling effective semi‑supervised learning. Experiments on MultiWOZ2.0/2.1 show SPACE improves success rates and BLEU scores by over 5% compared to prior methods.

Proton focuses on knowledge utilization. It introduces probing‑tuning, which extracts structured knowledge from pretrained models (e.g., via masked token predictions) and integrates it with task‑specific data. Applied to TableQA (Text‑to‑SQL) tasks such as Spider‑DK and Spider‑SYN, Proton achieves 6.9% and 16% improvements over state‑of‑the‑art baselines.

The article concludes that combining knowledge injection (SPACE) and knowledge extraction (Proton) substantially enhances pretrained dialogue models, and outlines future directions such as extending probing techniques to other domains and exploring more sophisticated dialogue strategies.