Online neural Q-Learning using heuristic weight assignment algorithm and optimization method

Abstract

The classic and almost recent robots still rely on fixed behavior based control. So the recent models of robots focus on increasing the robot`s ability to deal with any uncertainty from the environment. One approach of the paradigm is learning from experience and creating an appropriate control system from it. This approach is called Reinforcement Learning(RL). RL is a class of intelligent control methods that develop or improve the actions of the agent in an uncertain environment. By interacting with the environment, the agent learns and finds an optimal solution. To find the optimal solution, RL uses the value function. The value function is calculated using Bellman equation which is a nonlinear Lyapunov equation. But it usually requires knowledge of the system dynamics in order to solve for the value function. To avoid it, Q-Learning method for discrete space was introduced by Watkins. Another method is action dependent heuristic dynamic programming(AD HDP). AD HDP is based on an actor-critic structure that was introduced by Werbos. But the actor-critic structure involves training of two or more function approximators. This makes the training and the analysis of the results difficult. If it fails, it is unclear whether this is a result of the settings of the training parameters, the choice of function approximators or insufficient exploration in generating the data.

In contrast, Neural Q-Learning which involves the training of a function approximator was introduced by S. Hagen to apply Q-Learning for continuous space. This approach is based on Q-Learning for Linear Quadratic Regulation(LQR). But the learning time of Neural Q-Learning is very slow when it learns very complex systems such as Multi Input Multi Output (MIMO) system etc. Furthermore, batch learning cannot adapt in other environments without using a large data set for the training process. To solve these problems, we propose three contributions. First, we apply this learning to online learning t...