Title
Online Model-Free N-Step HDP with Stability Analysis
Abstract
Because of a powerful temporal-difference (TD) with λ [TD(λ)] learning method, this paper presents a novel n-step adaptive dynamic programming (ADP) architecture that combines TD(λ) with regular TD learning for solving optimal control problems with reduced iterations. In contrast with a backward view learning of TD(λ) that is required an extra parameter named eligibility traces to update at the end of each episode (offline training), the new design in this paper has forward view learning, which is updated at each time step (online training) without needing the eligibility trace parameter in various applications without mathematical models. Therefore, the new design is called the online model-free n-step action-dependent (AD) heuristic dynamic programming [NSHDP(λ)]. NSHDP(λ) has three neural networks: the critic network (CN) with regular one-step TD [TD(0)], the CN with n-step TD learning [or TD(λ)], and the actor network (AN). Because the forward view learning does not require any extra eligibility traces associated with each state, the NSHDP(λ) architecture has low computational costs and is memory efficient. Furthermore, the stability is proven for NSHDP(λ) under certain conditions by using Lyapunov analysis to obtain the uniformly ultimately bounded (UUB) property. We compare the results with the performance of HDP and traditional action-dependent HDP(λ) [ADHDP(λ)] with different λ values. Moreover, a complex nonlinear system and 2-D maze problem are two simulation benchmarks in this paper, and the third one is an inverted pendulum simulation benchmark, which is presented in the supplemental material part of this paper. NSHDP(λ) performance is examined and compared with other ADP methods.
Recommended Citation
S. Al-Dabooni and D. C. Wunsch, "Online Model-Free N-Step HDP with Stability Analysis," IEEE Transactions on Neural Networks and Learning Systems, Institute of Electrical and Electronics Engineers (IEEE), Jun 2019.
The definitive version is available at https://doi.org/10.1109/TNNLS.2019.2919614
Department(s)
Electrical and Computer Engineering
Keywords and Phrases
λ-Return; Action-Dependent (AD) Heuristic Dynamic Programming (ADHDP); Adaptive Dynamic Programming (ADP); Lyapunov Stability; Uniformly Ultimately Bounded (UUB)
International Standard Serial Number (ISSN)
2162-237X
Document Type
Article - Journal
Document Version
Citation
File Type
text
Language(s)
English
Rights
© 2019 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.
Comments
Article in Press