SPartAN: A Meta-algorithm for Reinforcement Learning using State Partitioning and Action Network

Abstract

Targeting finite-horizon Markov Decision Process problems, we propose a novel approach with an aim to significantly enhance the scalability of reinforcement learning (RL) algorithms. Our approach, which we call a State Partitioning and Action Network, SPartAN in short, is a meta-algorithm that offers a framework an RL algorithm can be incorporated into. Key ideas in SPartAN are threefold: reducing the size of an original RL problem by partitioning the state space into smaller compartments, using a simulation model to directly obtain values of the terminal states of the upstream compartment, and constructing a quality heuristic policy in the downstream compartment by an action network to use in the simulation. Using temporal difference learning as an example RL algorithm, we show that SPartAN is able to reliably derive a high quality policy solution. Through empirical analysis, we also find that a smaller downstream state subspace in SPartAN yields higher performance.