Memory & runtime efficient deep learning-based perception with prototypical encoder

Abstract

Perception is a crucial function for intelligent robots since most of the robots’ plans and actions should depend on the surrounding conditions. In recent years, deep learning-based robot perception has emerged as one of the most popular approaches, mostly due to its capability in generalizing to previously unseen but identical scenarios. While modern deep neural networks have shown extremely accurate performance in various downstream tasks, they often require tremendous computing resources. This effectiveness-efficiency tradeoff is an especially important issue in robot perception since the available computing resources are often limited, yet high accuracy and low latency are desired. In this thesis, we aim to address the effectiveness-efficiency tradeoff issue with the prototypical encoder, a lightweight neural network module that can be used to extract, transform, or generate new high-quality features in an efficient manner. A prototypical encoder operates by approximately modeling 2-ary interactions of elements in a set with a surrogate set of arbitrary size, which results in features with higher capacity compared to 1-ary functions. In addition, the computational complexity can be significantly reduced from conventional 2-ary functions by setting the surrogate set size to be smaller than the input set. We demonstrate the effectiveness and efficiency of prototypical encoders in two downstream applications: point cloud shape completion and multi-modal multi-task learning. In both applications, the prototypical encoder-based networks successfully outperform the baselines with a lower number of parameters and/or inference latency.