Why Natural Language Understanding Is Difficult: Structure Prediction, Semantic Representation, and Multimodal Context

In this article, Liu Zhiyuan, associate professor at Tsinghua University, discusses why natural language understanding (NLU) is challenging and offers a concise overview for readers interested in NLP.

NLU Is Essentially Structure Prediction

Natural language text is unstructured data composed of symbol sequences. Understanding requires predicting the underlying semantic structure, so tasks such as Chinese word segmentation, POS tagging, NER, coreference resolution, syntactic parsing, and semantic role labeling are all forms of structure prediction.

Different NLP tasks define different semantic structure spaces: text classification predicts a predefined label set, word segmentation predicts word boundaries, POS tagging predicts part‑of‑speech tags, etc.

The Key Is Semantic Representation

Early statistical methods used symbol‑based representations (e.g., bag‑of‑words, n‑grams) that ignore word order or deeper meaning. The deep‑learning era introduced distributed representations (embeddings), where each linguistic unit is encoded as a dense low‑dimensional vector, inspired by neural mechanisms.

Each entity is represented by a pattern of activity distributed over many computing elements, and each computing element is involved in representing many different entities.

Distributed representations enable semantic similarity calculations across words, phrases, sentences, and documents, but they suffer from limited interpretability, robustness, and transferability compared with human semantic processing.

Characteristics That Make Language Hard

Innovation : New words and new meanings constantly appear, expanding the semantic space and making exhaustive modeling difficult.

Recursion : Sentences can embed other sentences recursively, creating complex hierarchical structures that are hard for models to parse.

Ambiguity : Homophones and polysemy require contextual disambiguation at the character, word, phrase, and sentence levels.

Subjectivity : Individual experiences and cognitive differences lead to varied interpretations of the same text.

Sociality : Language reflects social context, status, and cultural norms; usage varies across formal reports, casual conversation, and different social groups.

Why NLU Is Difficult

The above properties create a complex, open, multimodal context that computers must model. Current semantic representation schemes are either too coarse (symbol‑based) or too task‑specific (embedding‑based), lacking the comprehensive, interpretable structure needed for true understanding.

Future work should explore hybrid representations that combine the generalization of embeddings with the modularity of symbolic structures, and build richer knowledge graphs (common sense, linguistic, world, cognitive, domain) to form a more powerful structured semantic space.

Multimodal Complex Context

Understanding language also requires external signals: textual context, prosody, visual cues, and other modalities. Disambiguating ambiguous units often depends on such multimodal information.

Conclusion

NLU difficulty arises from language’s innovative, recursive, ambiguous, subjective, and social nature. Progress requires better structured semantic representations, multimodal modeling, and integration of diverse knowledge sources so that computers can eventually comprehend language as humans do.

References

[1] Julia Hirschberg and Christopher D. Manning. Advances in Natural Language Processing. Science, 2015.

[2] Hinton, Geoffrey E., James L. McClelland, and David E. Rumelhart. Distributed Representations. Pittsburgh, PA: Carnegie‑Mellon University, 1984.

[3] Ferdinand de Saussure. Course in General Linguistics. Beijing: Commercial Press, 1980.

[4] Marc D. Hauser, Noam Chomsky, and W. Tecumseh Fitch. The Faculty of Language: What Is It, Who Has It, and How Did It Evolve?. Science, 2002.

[5] James W. Pennebaker. The Secret Life of Pronouns: What Our Words Say About Us. NY: Bloomsbury, 2011.

[6] Cristian Danescu‑Niculescu‑Mizil, Lillian Lee, Bo Pang, Jon Kleinberg. Echoes of Power: Language Effects and Power Differences in Social Interaction. WWW, 2012.

[7] Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. Semantics Derived Automatically from Language Corpora Contain Human‑like Biases. Science, 2017.

[8] George Lakoff. Don't Think of the Elephant. Zhejiang People’s Publishing House, 2013.