Adaptive Optimal Control of Unknown Constrained-Input Systems using Policy Iteration and Neural Networks
This paper presents an online policy iteration (PI) algorithm to learn the continuous-time optimal control solution for unknown constrained-input systems. The proposed PI algorithm is implemented on an actor-critic structure where two neural networks (NNs) are tuned online and simultaneously to generate the optimal bounded control policy. The requirement of complete knowledge of the system dynamics is obviated by employing a novel NN identifier in conjunction with the actor and critic NNs. It is shown how the identifier weights estimation error affects the convergence of the critic NN. A novel learning rule is developed to guarantee that the identifier weights converge to small neighborhoods of their ideal values exponentially fast. To provide an easy-to-check persistence of excitation condition, the experience replay technique is used. That is, recorded past experiences are used simultaneously with current data for the adaptation of the identifier weights. Stability of the whole system consisting of the actor, critic, system state, and system identifier is guaranteed while all three networks undergo adaptation. Convergence to a near-optimal control law is also shown. The effectiveness of the proposed method is illustrated with a simulation example.
H. Modares et al., "Adaptive Optimal Control of Unknown Constrained-Input Systems using Policy Iteration and Neural Networks," IEEE Transactions on Neural Networks and Learning Systems, vol. 24, no. 10, pp. 1513-1525, Institute of Electrical and Electronics Engineers (IEEE), Oct 2013.
The definitive version is available at http://dx.doi.org/10.1109/TNNLS.2013.2276571
Electrical and Computer Engineering
Keywords and Phrases
Adaptive Optimal Control; Input Constraints; Near-Optimal Control; Neural Networks (NNS); Optimal Control Solution; Optimal Controls; Persistence of Excitation; Simulation Example; Algorithms; Control; Neural Networks; Online Systems; Optimal Control Systems; Reinforcement Learning; Iterative Methods; Algorithm; Artificial Intelligence; Artificial Neural Network; Computer Simulation; Feedback System; Learning; Nonlinear System; Signal Processing; Theoretical Model; Feedback; Models; Theoretical; Neural Networks (Computer); Nonlinear Dynamics; Computer-Assisted; Optimal Control; Unknown Dynamics
International Standard Serial Number (ISSN)
Article - Journal
© 2013 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.