Modality Disentangled Learning for Incomplete Multimodal Emotion Recognition: A Primitive Memory Distillation Perspective

EMNLP 2026 Findings!

Jiaqi Zhang1, Zheng Pang1, Mengting Li1, Yiqi Wang2, Guangyuan Dong3, Chao Xue4, Yusen Wu5, Zihao Li6, Huy Phan7, Sicheng Zhao8, Björn W. Schuller9,10, Jiachen Luo9,11*

1Jiangsu University · 2Griffith University · 3National University of Singapore · 4University of New South Wales

5Fujian University of Technology · 6Xi'an Jiaotong-Liverpool University · 7German Research Center for Artificial Intelligence

8Tsinghua University · 9Technical University of Munich · 10Imperial College London · 11Queen Mary University of London

*Corresponding author.

Paper Code Conference BibTeX
Comparison of incomplete multimodal inputs, holistic recovery methods, and PriMD.
Figure 1. (a) Real-world inputs may suffer from partial modality missingness, leading to uncertain predictions. (b) Existing holistic generation, alignment, or distillation methods treat missing modalities as a whole. This tends to mix modality-shared semantics with uncertain modality-specific details, leaving the learned representations ambiguous. (c) Our PriMD first disentangles shared semantics from modality-specific information and performs constrained compensation for the missing modality-specific information. This enables more robust predictions under missing-modality conditions.

Our research focuses on the problem of missing modalities in multimodal emotion recognition. By examining existing approaches, we found that many methods generate, align, or distill a missing modality as a whole, overlooking the fact that different information components within a modality are not equally recoverable. Cross-modal shared emotional semantics can usually be reliably estimated from the available modalities, whereas modality-specific details are inherently more uncertain. Processing these two types of information together can produce ambiguous or even conflicting representations, leading to unstable predictions under different missing-modality conditions. Based on this insight, we propose PriMD, which reformulates the missing-modality problem as shared semantic estimation and constrained compensation of modality-specific information. PriMD first disentangles these two types of information, then organizes modality-specific information into discrete primitive memories and uses the shared semantics of the available modalities to dynamically retrieve relevant primitives. By avoiding unconstrained inference of the entire missing modality, PriMD enables more stable and robust emotion recognition.

Abstract

Multimodal Emotion Recognition (MER) systems often suffer from missing modalities in real-world scenarios. Existing methods usually generate, align, or distill missing modalities as a whole, overlooking the heterogeneous nature of the information carried by each modality. Such holistic treatment mixes inferable shared semantics with uncertain modality-specific details, yielding unstable representations and degrading robustness. To address this issue, we propose the Primitive Memory Distillation (PriMD) framework. Unlike existing methods, PriMD takes an intra-modal perspective and focuses on how different types of information within a modality differ in recoverability within each modality. PriMD first disentangles cross-modal shared semantics from modality-specific representations, and then discretizes the latter into learnable semantic primitives to construct modality-specific memory banks. When modalities are missing, PriMD is a teacher-student framework that the student model uses the shared semantics of available modalities as queries to dynamically retrieve primitives. It compensates for missing modality-specific information within a constrained memory space and aligns with the teacher model. Extensive experiments on IEMOCAP, CMU-MOSI, and CMU-MOSEI demonstrate that PriMD achieves state-of-the-art performance and consistently stronger robustness across a wide range of missing-modality settings, while mitigating the instability caused by holistic feature inference.

Method Overview

PriMD consists of a full-modality teacher model and an incomplete-modality student model. The teacher model first disentangles complete multimodal representations into cross-modal shared semantics and modality-specific information, and then constructs a discrete primitive memory based on the modality-specific information. In the student model, when some modalities are missing, the model dynamically retrieves relevant primitives according to the shared semantics of the available modalities to compensate for the missing modality-specific information. Finally, PriMD aligns the student model with the full-modality teacher model via retrieval-augmented distillation.

Overall framework of PriMD.
Figure 2. Overall framework of PriMD. The full-modality teacher model first encodes the inputs and disentangles shared semantics from modality-specific information via SSSD. Then, DPMC quantizes the modality-specific representations into discrete semantic primitives and constructs primitive memory banks. In missing-modality scenarios, the student model uses DRAD to dynamically retrieve relevant primitives from the memory banks based on the semantics of available modalities, enabling constrained compensation for the missing information. Finally, the compensated student model representations are aligned with the teacher model through retrieval-augmented distillation.
  • We revisit incomplete MER from an intra-modal perspective and formulate it as a unified process of shared-semantic estimation and constrained compensation of modality-specific information.
  • We design SSSD to disentangle shared and modality-specific features, and DPMC/DRAD to discretize the modality-specific space into semantic primitives and retrieve them for compensation under missing modalities.
  • Extensive experiments on three benchmarks show that PriMD outperforms prior methods under diverse missing-modality settings, with especially large gains when two modalities are absent.

Comparison with SOTA Methods

Inter-modal Missingness

Performance comparison with SOTA methods across six missing-modality combinations.
Table 1. Performance comparison with SOTA methods on two datasets under various missing-modality scenarios. "Avg." indicates the mean performance across all settings. The best and suboptimal results are highlighted. A, T, and V denote the acoustic, textual, and visual modalities, respectively. Gray shading indicates the presence of a modality, while gray dashed lines indicate its absence.

Intra-modal Missingness

Performance curves under varying missing rates on CMU-MOSI and CMU-MOSEI.
Figure 3. Performance curves of different methods under varying missing rates on the CMU-MOSI and CMU-MOSEI datasets. Subfigures (a)-(b) and (c)-(d) show F1 and MAE results on MOSI and MOSEI, respectively.

Ablation Study

w/o SSSD. Replacing the two projection branches with a single projection yields one entangled representation per modality and reduces average performance to 74.36% WA and 74.05% UA. Mixing shared and modality-specific information makes subsequent memory construction and distillation vulnerable to redundant or unstable modality-specific information.

w/o DPMC. Removing DPMC reduces average WA/UA to 75.95%/75.76%. This confirms that discrete primitive memory provides a constrained compensation space for uncertain missing modality-specific information.

w/o DRAD. Removing DRAD causes the largest degradation, to 72.31% WA and 72.10% UA. Representations from the available modalities alone cannot adapt to different missing-modality combinations.

Ablation results for SSSD, DPMC, and DRAD.
Table 2. Results of the ablation experiments under the Four-Class IEMOCAP. The best results are highlighted.

Computational Efficiency

PriMD reduces the parameter count, training time, and peak CUDA memory by 79.7%, 51.4%, and 82.2%, respectively, compared with HARDY-MER, while increasing inference latency by 8.1%.

Computational efficiency comparison between HARDY-MER and PriMD.
Table 4. Computational efficiency comparison.

Visualization Analysis

Fused Representations

Using only the text modality, the complete PriMD model produces a clearer class structure than its ablated variants: samples from the same class are more compact, while overlap between different classes is reduced.

t-SNE visualization of PriMD and its ablated variants.
Figure 7. t-SNE visualization of fused representations on Four-Class IEMOCAP using only the text modality.

Primitive Retrieval Patterns

Different emotion categories activate different subsets of primitives within each memory bank, while the audio, text, and visual banks learn distinct modality-specific primitive distributions.

Class-level primitive retrieval patterns for audio, text, and visual memory banks.
Figure 8. Visualization of class-level primitive retrieval patterns on audio, text, and visual memory banks.

BibTeX

@inproceedings{zhang2026primd,
  title     = {Modality Disentangled Learning for Incomplete Multimodal Emotion Recognition: A Primitive Memory Distillation Perspective},
  author    = {Jiaqi Zhang and
               Zheng Pang and
               Mengting Li and
               Yiqi Wang and
               Guangyuan Dong and
               Chao Xue and
               Yusen Wu and
               Zihao Li and
               Huy Phan and
               Sicheng Zhao and
               Bj{\"o}rn W. Schuller and
               Jiachen Luo},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026}
}