Hamiltonian-Driven Adaptive Dynamic Programming with Approximation Errors


In this article, we consider an iterative adaptive dynamic programming (ADP) algorithm within the Hamiltonian-driven framework to solve the Hamilton-Jacobi-Bellman (HJB) equation for the infinite-horizon optimal control problem in continuous time for nonlinear systems. First, a novel function, ``min-Hamiltonian,'' is defined to capture the fundamental properties of the classical Hamiltonian. It is shown that both the HJB equation and the policy iteration (PI) algorithm can be formulated in terms of the min-Hamiltonian within the Hamiltonian-driven framework. Moreover, we develop an iterative ADP algorithm that takes into consideration the approximation errors during the policy evaluation step. We then derive a sufficient condition on the iterative value gradient to guarantee closed-loop stability of the equilibrium point as well as convergence to the optimal value. A model-free extension based on an off-policy reinforcement learning (RL) technique is also provided. Finally, numerical results illustrate the efficacy of the proposed framework.


Electrical and Computer Engineering

Research Center/Lab(s)

Intelligent Systems Center

Publication Status

Early Access


This work was supported in part by the National Key Research and Development Program of China under Grant 2019YFB2102100; in part by the National Natural Science Foundation of China under Grant 61903028; in part by the Science and Technology Development Fund, Macao, under Grant 0015/2019/AKP; in part by the UM Macao Talent Programme under Grant UMMTP-2019-02; in part by the Guangdong–Hong Kong–Macao Joint Laboratory of Human-Machine Intelligence-Synergy Systems under Grant 2019B121205007; in part by the National Science Foundation under Grant CPS1851588 and Grant S&AS1849198; in part by the Mary K. Finley Missouri Endowment; in part by the Missouri S&T Intelligent Systems Center; in part by the Lifelong Learning Machines Program from DARPA/Microsystems Technology Office; and in part by the Army Research Laboratory under Agreement W911NF-18-2-0260.

Keywords and Phrases

Approximation Algorithms; Approximation Error; Costs; Dynamic Programming; Hamilton-Jacobi-Bellman (HJB) Equation; Hamiltonian-Driven Framework; Inexact Adaptive Dynamic Programming (ADP); Iterative Algorithms; Mathematical Model; Optimal Control; Stability Analysis

International Standard Serial Number (ISSN)

2168-2275; 2168-2267

Document Type

Article - Journal

Document Version


File Type





© 2021 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.

Publication Date

08 Sep 2021

PubMed ID