360 image upright adjustment via deep learning

Abstract

Spherical VR cameras can capture high-quality immersive VR images with a 360◦ field of view. However, in practice, when the camera orientation is not straight, the acquired VR image appears tilted when displayed on a VR headset, which diminishes the quality of the VR experience. To overcome this problem, we present a deep learning-based approach that can automatically estimate the orientation of a VR image and return its upright version. In contrast to existing methods, our approach does not require the presence of lines or horizon in the image, and thus can be applied on a wide range of scenes. rae> We first suggest a simple neural network architecture. This is composed of a CNN layer and a fully connected layers. After that we further investigate a better architecture that preserves spatial relationship of pixels in the 360 image. The graph convolutional layer is exploited to achieve the preservation. The latter version of network architecture demonstrates gain of accuracy without additional parameters nor gain of computation cost. Extensive experiments and comparisons with state-of-the-art methods have successfully confirmed the validity of our approach.