General‑Domain Conversational QA: Technologies, Challenges, and Alibaba UC’s Practice

General‑domain conversational QA, often referred to as intelligent assistants, has evolved from early rule‑based bots like ELIZA to modern platforms such as Siri, Watson, and Alibaba’s UC assistant. The article outlines this historical progression and positions current dialogue systems as the next generation of search engines.

Alibaba’s UC division, originating from the Chinese Yahoo search and later evolving through mobile and AI‑driven products (UC Browser, UC Headlines, Quark Smart Search), aims to build a cross‑platform intelligent assistant that integrates voice, search, recommendation, and NLP modules.

The dialogue platform is divided into three bot categories: TaskBot for task‑oriented interactions (e.g., weather, ticket booking), QABot for single‑turn question‑answering, and ChatBot for open‑ended conversations. Supporting these bots are diverse data sources, including knowledge graphs, user profiles, QA indexes, task‑dialogue libraries, and scripted scenarios, all orchestrated by a unified scheduling system.

Beyond basic NLU/NLG, the system incorporates Memory and Knowledge components to handle short‑term context and long‑term commonsense, as well as robot persona settings. Dialogue management transforms unstructured text into structured features, enabling policy layers that separate business logic from the engine for scalability and reuse.

Key technical challenges include achieving >90% accuracy for industrial deployment. To address data scarcity, the article discusses two major approaches: transfer learning—pre‑training embeddings on large corpora and fine‑tuning for specific tasks—and few‑shot learning, especially metric‑based methods that compare limited samples to known classes.

Multi‑dimensional dialogue management requires converting textual inputs into structured representations, applying decision policies, and ensuring extensibility through modular engine and policy layers. Advanced decision‑making techniques such as Bayesian networks, MDP, POMDP, and deep reinforcement learning are explored, though they demand extensive labeled data.

For QA, two paradigms are presented: knowledge‑graph‑based QA (KBQA) that maps entities to logical forms and queries graph databases, and DeepQA that extracts concise answers from unstructured web content, relying heavily on cross‑validation of multiple sources to ensure correctness.

Open‑domain chat combines retrieval‑based and generative approaches; the latter uses deep models (e.g., CVAE) to improve response diversity and relevance, enabling applications like intelligent commenting, poetry generation, and automatic couplet writing.