Full-body motion control of simulated 3D avatar from sparse sensor and direction control with deep reinforcement learning

Abstract

Interactive control of a 3D avatar is crucial to provide immersive experience for a VR user. Extensive research has been conducted on 3D avatar control techniques that allows a VR user to control a 3D avatar and interact with virtual objects while it moves around a large virtual environment. However, the research primarily focused on control system for a VR user who can physically walk around and there has been limited research on continuous control method for a seated VR user. In this work, the proposed method generates physically plausible full-body movement of a 3D avatar by combining GAN-based imitation learning with sparse sensor and directional control rewards. In this work, we propose a new reward function for sparse sensor tracking. Furthermore, we discuss how the value of the weights in this reward function affects the overall result of the generated motion. Our results show that a seated VR user can use sparse sensor signals and directional control inputs to realistically control a physic-based 3D avatar and interact with various virtual objects according to the laws of physics, e.g. blocking thrown projectiles, punching a target, or picking up an object.