Exploring complementary features in multi-modal speech emotion recognition

Speech emotion recognition (SER) is of great importance in human-computer interaction. Recent research has demonstrated that self-supervised learned acoustic and linguistic features are helpful in this task. However, few works have fully exploit the advantages of the pre-trained features in SER. The primary challenge is how to effectively extract the complementary emotional information implied in the pre-trained features of the respective modality. To tackle this challenge, we propose a novel modality-sensitive multimodal speech emotion recognition framework. In a nutshell, we aim to exploit the typical emotion features in each modality and then fuse the complementary emotional information for classification. Specifically, we first utilize the parallel uni-modal encoders to refine the emotion-related information from the pre-trained features of each modality. For better fusion of the multimodal features, we develop a group of learnable emotion query tokens to gather the emotional information from the refined acoustic and linguistic features with the cross-attention mechanism in the transformer decoder. Observing the modality bias problem in multimodal methods, we introduce the random modality masking training strategy to maximize the utilization of the emotional information in each modality and mitigate this problem. We evaluate our method on the widely used IEMOCAP dataset and achieve 1.1% and 0.9% improvements on the unweighted accuracy and weighted accuracy, respectively. Extensive experiments demonstrate the effectiveness of the proposed method.