Event-Driven Off-Policy Reinforcement Learning for Control of Interconnected Systems
Abstract
In this article, we introduce a novel approximate optimal decentralized control scheme for uncertain input-affine nonlinear-interconnected systems. In the proposed scheme, we design a controller and an event-triggering mechanism (ETM) at each subsystem to optimize a local performance index and reduce redundant control updates, respectively. To this end, we formulate a noncooperative dynamic game at every subsystem in which we collectively model the interconnection inputs and the event-triggering error as adversarial players that deteriorate the subsystem performance and model the control policy as the performance optimizer, competing against these adversarial players. To obtain a solution to this game, one has to solve the associated Hamilton-Jacobi-Isaac (HJI) equation, which does not have a closed-form solution even when the subsystem dynamics are accurately known. In this context, we introduce an event-driven off-policy integral reinforcement learning (OIRL) approach to learn an approximate solution to this HJI equation using artificial neural networks (NNs). We then use this NN approximated solution to design the control policy and event-triggering threshold at each subsystem. In the learning framework, we guarantee the Zeno-free behavior of the ETMs at each subsystem using the exploration policies. Finally, we derive sufficient conditions to guarantee uniform ultimate bounded regulation of the controlled system states and demonstrate the efficacy of the proposed framework with numerical examples.
Recommended Citation
V. Narayanan et al., "Event-Driven Off-Policy Reinforcement Learning for Control of Interconnected Systems," IEEE Transactions on Cybernetics, vol. 52, no. 3, pp. 1936 - 1946, Institute of Electrical and Electronics Engineers (IEEE), Mar 2022.
The definitive version is available at https://doi.org/10.1109/TCYB.2020.2991166
Department(s)
Electrical and Computer Engineering
Keywords and Phrases
Decentralized Control; Differential Game; Event-Triggered Learning; Off-Policy Learning
International Standard Serial Number (ISSN)
2168-2275; 2168-2267
Document Type
Article - Journal
Document Version
Citation
File Type
text
Language(s)
English
Rights
© 2022 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.
Publication Date
01 Mar 2022
PubMed ID
32639933
Comments
This work was supported in part by NSF under Grant ECCS 1406533 and Grant CMMI 1547042.