Unsupervised sim-real adaptation of the proprioceptive soft robot

Abstract

The skin of natural organisms makes them sense and detect their state and external stimuli without sight to unknown environments, helping to trigger behavioral responses. Inspired by this phenomenon, proprioceptive soft robots constructed from highly compliant materials are being highlighted these days as safer, more adaptable, and bio-inspired alternatives to traditional rigid robots. However, even with their capabilities of shape and behavioral changes and robustness to physical contact, their inherent hysteresis, nonlinearity and infinite degrees of freedom complicate high-level modeling and control. With the rise of learning-based control and perception in soft robotics to deal with those limitations, the sim-to-real transfer of soft robots has begun to be explored, which allows high-level control with efficient data collection. However, manufacturing variances, material fatigue, and high nonlinearities in soft sensors bring a gap between the modeled and physical sensors, which makes it hard to implement sensors in simulation. Taking inspiration from domain adaptation, we propose the sim-real synchronization method of soft robotic sensations by leveraging unsupervised domain-invariant representation learning. The dual cross-modal autoencoder structure is employed to reflect the common features in the domains into the shared latent space. In substitution with the direct mapping, the proposed approach enables data-efficient training by circumventing labeling procedures, and a generalized joint representation transferrable to multiple tasks. The framework is deployed into a popular soft robot design with a pneumatic multi-gait soft robot embedding eGaIn soft sensors. Reflecting the real-world deployments, the evaluation of the method proceeded with multi-task learning of kinematics estimation and collision detection on external contacts. This allows the robots to interact with their surroundings, either in simulation or the real world. Based on our proposed network, soft robots can be used for multimodal control even in simulation or reality, available for sim-to-real transfer. The results show that our network successfully synchronizes the sensor signals and detects the obstructions, providing soft robots with the data-driven multimodal control that was previously unachievable. We also demonstrated our method by implementing a reinforcement learning policy in simulation.