Classification and pose estimation of in-hand object through contact information

Abstract

In-hand object perception makes humans manipulate complicated shapes smoothly. It is also evident in robot manipulation systems. However, robots with vision sensors have difficulty reaching human-level dexterity due to a lack of sufficient tactile information channel as their human counterpart does. This work conducts classification and estimation of in-hand objects through contact information, and implements effective human tactile explorations strategy. First, with tactile sensors, contact information is extracted in the form of a point cloud that can simply and effectively represent the 3D shape of an object and utilized. Based on contact information in the corresponding point cloud format, this work implements PoinTacNet, a PointNet-based supervised learning architecture that can perform classification tasks. For PoinTacNet training, the contact information dataset is collected through simulation. Then, to reduce the domain gap between tactile information obtained in real and simulation, sim-to-real transfer learning is applied to perform classification tasks through real contact information. With implementation of effective human tactile exploration through class likelihood distribution, this work shows the improvements in classification certainty increases. The pose estimation tasks are performed on the classified objects through point cloud registration algorithm. This work provided quantitative classification accuracy and pose estimation results for 10 objects obtained from the McMaster public dataset. This work reduced the domain gap of contact information between simulation and real-world through sim-to-real transfer learning, and the results showed an accuracy of about 83.89% using real-world contact information. Also, real-to-real learning with data augmentation techniques that PoinTaCNet, trained via real tactile data, has difficulty in obtaining good classification performance, and successfully learns and classifies small amounts of real contact information through simulation and transfer learning. With the implementation of the re-grasping scenario, which is an effective human exploration strategy, this work demonstrated improvement of quantitative indicators on accuracy of classification and the performance. Using a point cloud registration algorithm called FilterReg, this work was able to derive an average RMSE value of about 5.032º.