BIMODAL FUSION NETWORK FOR BASIC TASTE SENSATION RECOGNITION FROM ELECTROENCEPHALOGRAPHY AND ELECTROMYOGRAPHY

Taste sensation can be objectively measured using electroencephalography (EEG) or electromyography (EMG). However, it is still challenging to effectively utilize the complementary information from EEG and EMG signals in taste sensation recognition. This paper proposes a bimodal fusion network (Bi-FusionNet) for recognizing basic taste sensations (sour, sweet, bitter, salty, umami, and blank). Two convolutional backbones with similar structures are designed to separately extract the single-modal features of EEG and EMG. Then, EEG and EMG features are concatenated for bimodal interaction and complementarity. Finally, three loss functions are adopted: a center loss for aggregating intra-class samples, a mean squared error loss for sequence positions for minimizing the difference between signals during the stimulation, and a softmax loss for minimizing the entropy of prediction and labels. The results on the taste sensation dataset show that bimodal fusion improves recognition performance, and Bi-FusionNet outperforms single-modal methods and other fusion methods. Bi-FusionNet paves the way for the application of multimodal fusion in taste sensation recognition.