Optimal Tracking Control of Uncertain Systems: On-Policy and Off-Policy Reinforcement Learning Approaches


This chapter presents adaptive solutions to the optimal tracking problem of nonlinear discrete-time and continuous-time systems. These methods find the optimal feedback and feedforward parts of the control input simultaneously, without requiring complete knowledge of the system dynamics. First, an augmented system composed of the error system dynamics and the reference trajectory dynamics is formed to introduce a new nonquadratic discounted performance function for the optimal tracking control problem. This encodes the input constrains caused by the actuator saturation into the optimization problem. Then, the tracking Bellman equation and the tracking Hamilton-Jacobi-Bellman equation for both discrete-time and continuous-time systems are derived. Finally, to obviate the requirement of complete knowledge of the system dynamics in finding the Hamilton-Jacobi-Bellman solution, integral reinforcement learning and off-policy reinforcement learning algorithms are developed for continuous-time systems, and a reinforcement learning algorithm on an actor-critic structure is developed for discrete-time systems.


Electrical and Computer Engineering

Keywords and Phrases

Digital Control Systems; Discrete Time Control Systems; Dynamic Programming; Learning Algorithms; Navigation; Nonlinear Systems; Optimization; Reinforcement Learning; System Theory; Actor Critic; Discrete - Time Systems; Hamilton Jacobi Bellman; Hamilton Jacobi Bellman Equation; Nonlinear Discrete-Time; Optimal Tracking Control; Reference Trajectories; Reinforcement Learning Approach; Continuous Time Systems; Actor-Critic; Integral Reinforcement Learning; Off-Policy Reinforcement Learning

International Standard Book Number (ISBN)

978-0128054376; 978-0128052464

Document Type

Book - Chapter

Document Version


File Type





© 2016 Elsevier, All rights reserved.

Publication Date

01 Jan 2016