Deep neural mismatch model for VR sickness assessment in virtual environment

Abstract

VR sickness is one of the main bottlenecks for the proliferation of VR market. Since VR sickness causes several physical symptoms such as burping, headache, and dizziness to viewers, it is required to automatically predict the level of VR sickness of VR content. VR sickness in human perception can be explained by neural mismatch model. According to neural mismatch model, discrepancy between input sensory signals by sensed by our receptors (e.g., eyes for visual signals) and the expected sensory signals predicted by our neural store with the past experience leads to VR sickness. In this paper, we propose a novel objective VR sickness assessment (VRSA) method based on neural mismatch model for 360-degree video. The proposed VRSA framework is composed of three deep networks which are deep neural store network, deep comparison network, and deep VR sickness score prediction network. The proposed neural store network is to expect the visual sensory information. In this paper, to teach the deep neural store network the general experience of people, the neural store network is trained with normal videos including non-exceptional motion pattern or high frame rate. The deep comparison network is to encode the neural mismatch feature from the visual sensory input sensed by our eyes and the expected visual sensory information predicted by our neural store network. Finally, the level of VR sickness is predicted by the VR sickness score prediction from the neural mismatch feature. In particular, we built eighty 360-degree videos with four different frame rates and conducted extensive subjective experiments to obtain physiological signals and subjective questionnaire for evaluating VR sickness. In our experiment, the prediction performance of the proposed VRSA was verified on two 360-dgree video databases for VRSA, which have different factors of VR sickness (motion and frame rate). Experimental results demonstrated that the proposed VRSA method considering neural mismatch model achieved a high correlation with human subjective score for VR sickness.