6D2DF: 6D object pose tracking using 2D detection-based filtering for synchronization in digital twin

Abstract

We propose a system that tracks category-level 6D object pose and updates the virtual replica in the digital twin corresponding to the real-world object. Recent studies on 6D pose tracking proposed deep-learning-based networks to estimate and track the 6D pose using keypoints or features extracted from RGB and depth images. However, multi-stage pipelines, including detection and refinement stages, have a long inference time for real-time 6D pose tracking. For this reason, we utilize a single-shot network, CenterSnap, to achieve fast 6D pose tracking for category-level multiple objects. CenterSnap extracts the 6D pose and 3D sizes and generates a 3D bounding box from them. We use the keypoints of the 3D bounding box by applying the optimized Kalman filter and Euclidean distance algorithm to jointly track the 6D pose. In addition, we propose 2D detector-based filtering that removes the false positive predictions and corrects the classification results of the 6D object pose tracker. We conduct the 2D object tracking using YOLOv7, Kalman filter, and Hungarian algorithm. The Intersection over Union and center distance are calculated between the 2D bounding boxes from the 6D pose tracking module and 2D tracking module. The tracked objects are filtered according to the degree of match of the object between the two modules. By comparing with the baseline, CeterSnap, our proposed system reduces the false positives through 2D detector-based filtering and increases true positives by tracking the 6D pose. 3D CAD models for each category are assigned as digital twins for real objects. Those models are created or removed in the Unity project, corresponding to the 6D pose tracking results. The tracked 6D poses are transmitted through the UDP server and transformed into the rotation and position value of the game object in Unity. Therefore, the real objects are synchronized with the virtual objects in the digital twin.