Tree search in partially observable markovian decision process with preference learning for object manipulation

Abstract

To be deployed in a household environment to assist a human, a robot must be able to make decisions under partial observability. A robot can cope with the uncertainty of its observation by making sensing actions such as clearing an object in front of a shelf that occludes the target behind and estimating its shape. Partially Observable Markovian Decision Process (POMDP) is the principled framework that enables the robot to perform such information-gathering actions. However, robot manipulation domains involve high-dimensional and continuous observation and action spaces, yet most POMDP solvers have been limited to discrete spaces. Recently, POMCPOW, a continuous POMDP solver that resorts to sampling and progressive widening, has been proposed, yet it is too slow to be practical in robot manipulation problems that incorporate camera observations and multiple objects. To facilitate a more efficient search, we propose a framework that learns to guide the search from past planning experience. Our method specifically adopts preference learning that can leverage both success and failure trajectories to learn effective heuristics for search even with a small number of past experiences. We demonstrate the efficacy of our framework in several continuous partially observable robotics domains, such as light-dark room domain and real-world robot manipulation.