On-policy Approximate Dynamic Programming for Optimal Control of non-linear systems

Optimal control theory deals with finding the policy that minimizes the discounted infinite horizon quadratic cost function. For finding the optimal control policy, the solution of the Hamilton-Jacobi-Bellman (HJB) equation must be found i.e. the value function which satisfies the Bellman equation. However, the HJB is a partial differential equation that is difficult to solve for a nonlinear system. The paper employs the approximate dynamic programming method to solve the HJB equation for the deterministic nonlinear discrete-time systems in continuous state and action space. The approximate solution of the HJB is found by the policy iteration algorithm which has the framework of actor-critic architecture. The control policy and value function are approximated using function approximators such as neural network represented in the form of linearly independent basis function. The gradient descent optimization algorithm is employed to tune the weights of the actor and critic network. The control algorithm is implemented for cart pole inverted pendulum system, the effectiveness of this approach is provided in simulations.