An Improved N-Step Value Gradient Learning Adaptive Dynamic Programming Algorithm for Online Learning
Abstract
In problems with complex dynamics and challenging state spaces, the dual heuristic programming (DHP) algorithm has been shown theoretically and experimentally to perform well. This was recently extended by an approach called value gradient learning (VGL). VGL was inspired by a version of temporal difference (TD) learning that uses eligibility traces. The eligibility traces create an exponential decay of older observations with a decay parameter (λ). This approach is known as TD(λ), and its DHP extension is known as VGL(λ), where VGL(0) is identical to DHP. VGL has presented convergence and other desirable properties, but it is primarily useful for batch learning. Online learning requires an eligibility-trace-work-space matrix, which is not required for the batch learning version of VGL. Since online learning is desirable for many applications, it is important to remove this computational and memory impediment. This paper introduces a dual-critic version of VGL, called N-step VGL (NSVGL), that does not need the eligibility-trace-work-space matrix, thereby allowing online learning. Furthermore, this combination of critic networks allows an NSVGL algorithm to learn faster. The first critic is similar to DHP, which is adapted based on TD(0) learning, while the second critic is adapted based on a gradient of n-step TD(λ) learning. Both networks are combined to train an actor network. The combination of feedback signals from both critic networks provides an optimal decision faster than traditional adaptive dynamic programming (ADP) via mixing current information and event history. Convergence proofs are provided. Gradients of one- and n-step value functions are monotonically nondecreasing and converge to the optimum. Two simulation case studies are presented for NSVGL to show their superior performance.
Recommended Citation
S. Al-Dabooni and D. C. Wunsch, "An Improved N-Step Value Gradient Learning Adaptive Dynamic Programming Algorithm for Online Learning," IEEE Transactions on Neural Networks and Learning Systems, vol. 31, no. 4, pp. 1155 - 1169, Institute of Electrical and Electronics Engineers (IEEE), Apr 2020.
The definitive version is available at https://doi.org/10.1109/TNNLS.2019.2919338
Department(s)
Electrical and Computer Engineering
Research Center/Lab(s)
Intelligent Systems Center
Second Research Center/Lab
Center for High Performance Computing Research
Keywords and Phrases
Adaptive Dynamic Programming (ADP); Convergence Analysis; Eligibility Traces; Online Learning; Reinforcement Learning; Temporal Difference (TD); Value Gradient Learning (VGL)
International Standard Serial Number (ISSN)
2162-237X
Document Type
Article - Journal
Document Version
Citation
File Type
text
Language(s)
English
Rights
© 2019 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.
Publication Date
01 Apr 2020
Comments
This work was supported in part by the Missouri University of Science and Technology Intelligent Systems Center, in part by the Mary K. Finley Missouri Endowment, in part by the National Science Foundation, in part by the Lifelong Learning Machines Program from the DARPA/Microsystems Technology Office, in part by the Army Research Laboratory (ARL), in part by the Cooperative Agreement under Grant W911NF-18-2-0260.