Hamiltonian-Driven Adaptive Dynamic Programming with Approximation Errors
Abstract
In this article, we consider an iterative adaptive dynamic programming (ADP) algorithm within the Hamiltonian-driven framework to solve the Hamilton-Jacobi-Bellman (HJB) equation for the infinite-horizon optimal control problem in continuous time for nonlinear systems. First, a novel function, ``min-Hamiltonian,'' is defined to capture the fundamental properties of the classical Hamiltonian. It is shown that both the HJB equation and the policy iteration (PI) algorithm can be formulated in terms of the min-Hamiltonian within the Hamiltonian-driven framework. Moreover, we develop an iterative ADP algorithm that takes into consideration the approximation errors during the policy evaluation step. We then derive a sufficient condition on the iterative value gradient to guarantee closed-loop stability of the equilibrium point as well as convergence to the optimal value. A model-free extension based on an off-policy reinforcement learning (RL) technique is also provided. Finally, numerical results illustrate the efficacy of the proposed framework.
Recommended Citation
Y. Yang et al., "Hamiltonian-Driven Adaptive Dynamic Programming with Approximation Errors," IEEE Transactions on Cybernetics, Institute of Electrical and Electronics Engineers (IEEE), Sep 2021.
The definitive version is available at https://doi.org/10.1109/TCYB.2021.3108034
Department(s)
Electrical and Computer Engineering
Research Center/Lab(s)
Intelligent Systems Center
Publication Status
Early Access
Keywords and Phrases
Approximation Algorithms; Approximation Error; Costs; Dynamic Programming; Hamilton-Jacobi-Bellman (HJB) Equation; Hamiltonian-Driven Framework; Inexact Adaptive Dynamic Programming (ADP); Iterative Algorithms; Mathematical Model; Optimal Control; Stability Analysis
International Standard Serial Number (ISSN)
2168-2275; 2168-2267
Document Type
Article - Journal
Document Version
Citation
File Type
text
Language(s)
English
Rights
© 2021 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.
Publication Date
08 Sep 2021
PubMed ID
34495864
Comments
This work was supported in part by the National Key Research and Development Program of China under Grant 2019YFB2102100; in part by the National Natural Science Foundation of China under Grant 61903028; in part by the Science and Technology Development Fund, Macao, under Grant 0015/2019/AKP; in part by the UM Macao Talent Programme under Grant UMMTP-2019-02; in part by the Guangdong–Hong Kong–Macao Joint Laboratory of Human-Machine Intelligence-Synergy Systems under Grant 2019B121205007; in part by the National Science Foundation under Grant CPS1851588 and Grant S&AS1849198; in part by the Mary K. Finley Missouri Endowment; in part by the Missouri S&T Intelligent Systems Center; in part by the Lifelong Learning Machines Program from DARPA/Microsystems Technology Office; and in part by the Army Research Laboratory under Agreement W911NF-18-2-0260.