Recently, ACM Multimedia 2022 was successfully held in Lisbon, Portugal. Li Rui, a PhD candidate co-supervised by Prof. Bao-Liang Lu and Associate Professor Wei-Long Zheng from Key Laboratory of Shanghai Commission for Intelligent Interaction and Cognitive Engineering, Department of Computer Science and Engineering, won the ACM Multimedia Top paper Award.

ACM Multimedia (CCF A) is the worldwide premier conference and a key world event to display scientific achievements and innovative industrial products in the multimedia field. ACM Multimedia 2022 has a total of 2473 valid submissions, with an acceptance rate of 27.9%. A total of 13 Top papers were selected, including one Best Paper and one Best Student Paper.

Background

Affective Brain-computer Interface has achieved considerable advances that researchers can successfully interpret labeled and flawless EEG data collected in laboratory settings. However, the annotation of EEG data is time-consuming and daily collected EEG data may be partially damaged since EEG signals are sensitive to noise. These challenges limit the application of EEG in practical scenarios.

Introduction

The title of the paper is: ‘A Multi-view Spectral-Spatial-Temporal Masked Autoencoder for Decoding Emotions with Self-supervised Learning’. In this paper, they proposed a Multi-view Spectral-Spatial-Temporal Masked Autoencoder (MV-SSTMA) with self-supervised learning to tackle the realistic challenges of EEG towards daily applications. The MV-SSTMA is based on a multi-view CNN-Transformer hybrid structure, interpreting the emotion-related knowledge of EEG signals from spectral, spatial, and temporal perspectives. The model consists of three stages: 1) In the generalized pre-training stage, channels of unlabeled EEG data from all subjects are randomly masked and later reconstructed to learn the generic representations from EEG data; 2) In the personalized calibration stage, only few labeled data from a specific subject are used to calibrate the model; 3) In the personal test stage, our model can decode personal emotions from the sound EEG data as well as damaged ones with missing channels. Extensive experiments on two open emotional EEG datasets demonstrate that their proposed model achieves state-of-the-art performance on emotion recognition. In addition, under the abnormal circumstance of missing channels, the proposed model can still effectively recognize emotions.

The overall process of the proposed method.

Paper link：