Title

Online Model-Free N-Step HDP with Stability Analysis

Abstract

Because of a powerful temporal-difference (TD) with λ [TD(λ)] learning method, this paper presents a novel n-step adaptive dynamic programming (ADP) architecture that combines TD(λ) with regular TD learning for solving optimal control problems with reduced iterations. In contrast with a backward view learning of TD(λ) that is required an extra parameter named eligibility traces to update at the end of each episode (offline training), the new design in this paper has forward view learning, which is updated at each time step (online training) without needing the eligibility trace parameter in various applications without mathematical models. Therefore, the new design is called the online model-free n-step action-dependent (AD) heuristic dynamic programming [NSHDP(λ)]. NSHDP(λ) has three neural networks: the critic network (CN) with regular one-step TD [TD(0)], the CN with n-step TD learning [or TD(λ)], and the actor network (AN). Because the forward view learning does not require any extra eligibility traces associated with each state, the NSHDP(λ) architecture has low computational costs and is memory efficient. Furthermore, the stability is proven for NSHDP(λ) under certain conditions by using Lyapunov analysis to obtain the uniformly ultimately bounded (UUB) property. We compare the results with the performance of HDP and traditional action-dependent HDP(λ) [ADHDP(λ)] with different λ values. Moreover, a complex nonlinear system and 2-D maze problem are two simulation benchmarks in this paper, and the third one is an inverted pendulum simulation benchmark, which is presented in the supplemental material part of this paper. NSHDP(λ) performance is examined and compared with other ADP methods.

Department(s)

Electrical and Computer Engineering

Comments

Article in Press

Keywords and Phrases

λ-Return; Action-Dependent (AD) Heuristic Dynamic Programming (ADHDP); Adaptive Dynamic Programming (ADP); Lyapunov Stability; Uniformly Ultimately Bounded (UUB)

International Standard Serial Number (ISSN)

2162-237X

Document Type

Article - Journal

Document Version

Citation

File Type

text

Language(s)

English

Rights

© 2019 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.

Publication Date

01 Jun 2019

Link to Full Text

Share

 
COinS