Multimodal Text and Speech Emotion Analysis: Overview, MSCNN‑SPU Model, and Domain Adaptation

The presentation begins with a brief history of emotion theory and explains why sentiment analysis based solely on text is insufficient for many real‑world scenarios, motivating the use of both textual and acoustic modalities.

It then surveys single‑modal text approaches, ranging from classic sentiment lexicons to neural models such as TextCNN, BiLSTM, TextRCNN, FastText, DAN, and large pre‑trained Transformers, highlighting their strengths and limitations for short versus long texts.

Next, the audio side is examined. Fundamental signal processing steps (A/D conversion, framing, windowing) produce features like spectrograms, mel‑spectrograms, and MFCCs. Early methods used handcrafted low‑level descriptors and Bag‑of‑Audio‑Words, while later work applied CNNs (e.g., AlexNet on spectrograms), TextCNN‑style convolutions, and hybrid CNN‑LSTM architectures, often enhanced with attention mechanisms.

The talk categorises multimodal fusion strategies into early, late, and multi‑level fusion, and discusses simple concatenation versus attention‑based (including multi‑hop) fusion. Common datasets such as IEMOCAP are introduced, noting the typical four‑class setup (neutral, angry, sad, excited).

Central to the session is the MSCNN‑SPU model. Inspired by the short‑text nature of conversational data and the global acoustic patterns of speech, the model combines shallow multi‑scale CNNs (MSCNN) with a statistical pooling unit (SPU) that outputs mean, max, and standard‑deviation statistics. Text features include SWEM‑max/avg embeddings, while audio features incorporate MFCCs and speaker‑identity X‑vectors extracted by a pre‑trained TDNN. An attention layer uses audio‑level pooled vectors as queries to attend over textual local features, and the final representation concatenates attention outputs, MSCNN‑SPU embeddings, X‑vectors, and SWEM vectors (≈1024‑dimensional) before classification.

Experimental results on IEMOCAP show that MSCNN‑SPU surpasses state‑of‑the‑art baselines by 3–4 % weighted accuracy, with ablation studies confirming the importance of attention, SPU, and speaker embeddings. Replacing MSCNN‑SPU components with BiLSTM degrades performance significantly.

The second part addresses unsupervised domain adaptation for deploying models across heterogeneous business domains (e.g., insurance, education, banking). It outlines two families of methods: BatchNorm statistics alignment and adversarial training where a domain discriminator forces source and target feature distributions to converge. The workflow involves supervised pre‑training on labelled source data, copying the feature extractor to the target domain, and alternating discriminator and feature‑encoder updates.

Business case studies demonstrate that multimodal models improve customer‑service quality monitoring (e.g., from 89 % to 94 % accuracy in binary sentiment tasks) and that adversarial domain adaptation can raise target‑domain accuracy by 2–4 % compared with naïve transfer.

Finally, the speaker outlines future directions: incorporating fine‑grained temporal alignment between speech and text, leveraging robust pre‑trained audio encoders (Wav2Vec, CPC, APC), exploring large‑scale multimodal Transformers for early fusion, and extending the framework to additional modalities such as video.

A Q&A session covers practical concerns about noise, feature dimensionality, multimodal interaction, transformer alternatives, speaker‑identification usage, and deployment efficiency.