Improving multi-modal emotion recognition with counterpart data in dyadic conversations

Abstract

Today, as we live in numerous interactions, many studies have tried to predict human emotions. Since our daily life consists of countless interactions, it is better to predict human emotions between interactions. However, most studies have focused only on the speaker's data, not the counterpart's data, to predict the speaker's emotions because datasets which labeled human emotions in the naturalistic conversation are rare. In this study, we propose a method for predicting the emotions of the speaker in the naturalistic conversation using a speaker encoder and counterpart encoder composed of CNN-LSTM deep learning networks. We used emotion-related data called K-EmoCon collected during the debate process to empirically evaluate our model. The results showed that the counterpart's speech and the physiological signals had a positive impact on predicting the speaker's emotions. Through this paper, we hope to be helpful in the study of predicting emotions in naturalistic conversation.