Optimal Tracking Control of Uncertain Systems: On-Policy and Off-Policy Reinforcement Learning Approaches
Abstract
This chapter presents adaptive solutions to the optimal tracking problem of nonlinear discrete-time and continuous-time systems. These methods find the optimal feedback and feedforward parts of the control input simultaneously, without requiring complete knowledge of the system dynamics. First, an augmented system composed of the error system dynamics and the reference trajectory dynamics is formed to introduce a new nonquadratic discounted performance function for the optimal tracking control problem. This encodes the input constrains caused by the actuator saturation into the optimization problem. Then, the tracking Bellman equation and the tracking Hamilton-Jacobi-Bellman equation for both discrete-time and continuous-time systems are derived. Finally, to obviate the requirement of complete knowledge of the system dynamics in finding the Hamilton-Jacobi-Bellman solution, integral reinforcement learning and off-policy reinforcement learning algorithms are developed for continuous-time systems, and a reinforcement learning algorithm on an actor-critic structure is developed for discrete-time systems.
Recommended Citation
B. Kiumarsi et al., "Optimal Tracking Control of Uncertain Systems: On-Policy and Off-Policy Reinforcement Learning Approaches," Control of Complex Systems: Theory and Applications, pp. 165 - 186, Elsevier, Jan 2016.
The definitive version is available at https://doi.org/10.1016/B978-0-12-805246-4.00005-7
Department(s)
Electrical and Computer Engineering
Keywords and Phrases
Digital Control Systems; Discrete Time Control Systems; Dynamic Programming; Learning Algorithms; Navigation; Nonlinear Systems; Optimization; Reinforcement Learning; System Theory; Actor Critic; Discrete - Time Systems; Hamilton Jacobi Bellman; Hamilton Jacobi Bellman Equation; Nonlinear Discrete-Time; Optimal Tracking Control; Reference Trajectories; Reinforcement Learning Approach; Continuous Time Systems; Actor-Critic; Integral Reinforcement Learning; Off-Policy Reinforcement Learning
International Standard Book Number (ISBN)
978-0128054376; 978-0128052464
Document Type
Book - Chapter
Document Version
Citation
File Type
text
Language(s)
English
Rights
© 2016 Elsevier, All rights reserved.
Publication Date
01 Jan 2016
