Sparse Online Kernelized Actor-Critic Learning in Reproducing Kernel Hilbert Space
Abstract
In this paper, we develop a novel non-parametric online actor-critic reinforcement learning (RL) algorithm to solve optimal regulation problems for a class of continuous-time affine nonlinear dynamical systems. To deal with the value function approximation (VFA) with inherent nonlinear and unknown structure, a reproducing kernel Hilbert space (RKHS)-based kernelized method is designed through online sparsification, where the dictionary size is fixed and consists of updated elements. In addition, the linear independence check condition, i.e., an online criteria, is designed to determine whether the online data should be inserted into the dictionary. The RHKS-based kernelized VFA has a variable structure in accordance with the online data collection, which is different from classical parametric VFA methods with a fixed structure. Furthermore, we develop a sparse online kernelized actor-critic learning RL method to learn the unknown optimal value function and the optimal control policy in an adaptive fashion. The convergence of the presented kernelized actor-critic learning method to the optimum is provided. The boundedness of the closed-loop signals during the online learning phase can be guaranteed. Finally, a simulation example is conducted to demonstrate the effectiveness of the presented kernelized actor-critic learning algorithm.
Recommended Citation
Y. Yang et al., "Sparse Online Kernelized Actor-Critic Learning in Reproducing Kernel Hilbert Space," Artificial Intelligence Review, vol. 55, pp. 23 - 58, Springer, Jan 2022.
The definitive version is available at https://doi.org/10.1007/s10462-021-10045-9
Department(s)
Electrical and Computer Engineering
Research Center/Lab(s)
Intelligent Systems Center
Keywords and Phrases
Actor-Critic Learning; Non-Parametric Learning; Online Sparsification; Reproducing Kernel Hilbert Space; Value Function Approximation
International Standard Serial Number (ISSN)
1573-7462; 0269-2821
Document Type
Article - Journal
Document Version
Citation
File Type
text
Language(s)
English
Rights
© 2022 The Author(s), under exclusive licence to Springer Nature B.V., All rights reserved.
Publication Date
01 Jan 2022
Comments
This work was supported in part by the National Natural Science Foundation of China under Grants 61903028, 61973330, 61803371 and 61773075, in part by the Beijing Natural Science Foundation under Grant 212038, in part by the Open Research Project of the State Key Laboratory of Management and Control for Complex Systems, Institute of Sciences under Grant 20210108, in part by the Open Research Project of the State Key Laboratory of Industrial Control Technology, Zhejiang University, China under Grant ICT2021B48, in part by the Fundamental Research Funds for the Central Universities under Grant 2019NTST25, and in part by the State Key Laboratory of Synthetical Automation for Process Industries under Grant 2019-KF-23-03.