Modeling human affective states by collecting mobile multimodal datasets in the wild

Abstract

Recent studies in emotional intelligence or affective computing have a great interest in interpreting human behaviors and mental health by leveraging mobile and wearable sensor datasets from daily contexts. For this, it is crucial to build and make available for public use such large-scale and in-the-wild datasets that include an appropriate number of affective state labels. However, there is a lack of open datasets collected in real-world contexts with affective and cognitive state labels such as emotion, stress, and attention.

In this work, we first validate the self-reported emotion labels obtained via the mobile experience sampling method~(ESM) that is commonly used to measure people's affective states by randomly requesting self-report responses. Next, we release an initial in-the-wild naturalistic dataset of smartphone use, wearable sensing, and self-reported affect states. We supplement three additional datasets and examine the cross-dataset generalizability among them. Finally, we discuss the implications and contributions of the collected datasets that can help to advance the research and development of affective computing, emotional intelligence, and attention management domains.