Full-body avatar motion generation and visual guidance for interaction in remote telepresence between dissimilar indoor environments

Abstract

This paper presents two key technologies for mixed reality (MR) telepresence that allow geographically distant users to feel as if they exist in the same space through virtual avatars, and to remotely interact using verbal/nonverbal communication. The first is a deep neural network model designed to generate full-body motion from sparse data collected by commercial MR devices. This model is used to animate remote virtual avatars that mimic the user's movements in real-time. The second is the visual guidance method that recommends placements (location, direction) of user space appropriate for remote interaction. This technique allows seamless remote communication by overcoming the difference in size and arrangement of the two remote spaces. Based on the interaction features definition of placements and the similarity measure between them, our method calculates and visualizes the recommendation score of each placement in the user space for predefined interaction targets. As the user moves to a certain location for interaction guided by the visuals, the system moves the avatar to the corresponding remote space placement, enabling accurate gaze and pointing interaction in the remote telepresence environment.