Speaker-Invariant Affective Representation Learning Via Adversarial Training

Representation learning for speech emotion recognition is challenging due to labeled data sparsity issue and lack of gold-standard references. In addition, there is much variability from input speech signals, human subjective perception of the signals and emotion label ambiguity. In this paper, we propose a machine learning framework to obtain speech emotion representations by limiting the effect of speaker variability in the speech signals. Specifically we propose to disentangle the speaker characteristics from emotion through an adversarial training network in order to better represent emotion. Our method combines the gradient reversal technique with an entropy loss function to remove such speaker information. We evaluate our approach on IEMOCAP and CMU-MOSEI datasets to provide across-corpora evaluation and show this speaker-information removal can improve speech emotion classification.