3D Hand Pose Estimation with a Single Infrared Camera via Domain Transfer Learning

Abstract

Previous methods successfully estimated 3D hand poses from unblurred depth images with slow and smooth hand motions. However, the performance drops when the depth images are contaminated by motion blur due to fast hand motion. In this paper, we exploit an infrared (IR) image input, which is weakly blurred under fast hand motion. The proposed method is based on domain transfer learning from depth to infrared images. Note we do not have IR images with hand skeletons, thus proposing self-supervision rather than direct supervision using the skeleton labels. We train a Hand Image Generator (HIG) and two Hand Pose Estimators (HPEs) on paired depth and infrared images via self-supervision using a consistency loss, guided by an existing HPE trained on paired depth and hand skeleton entries. The IR-based HPE is then refined on the weakly blurred infrared images. The qualitative and quantitative experiments demonstrate that the proposed method accurately estimates 3D hand poses under motion blur by fast hand motion, while existing depth-based methods fail. Our solution therefore supports fast 3D manipulation of virtual objects for augmented reality applications. Our model and dataset are publicly available for future research.(1)