Event-Driven Off-Policy Reinforcement Learning for Control of Interconnected Systems

Author

Vignesh Narayanan
Hamidreza Modares
Sarangapani Jagannathan, Missouri University of Science and TechnologyFollow
Frank L. Lewis

Abstract

In this article, we introduce a novel approximate optimal decentralized control scheme for uncertain input-affine nonlinear-interconnected systems. In the proposed scheme, we design a controller and an event-triggering mechanism (ETM) at each subsystem to optimize a local performance index and reduce redundant control updates, respectively. To this end, we formulate a noncooperative dynamic game at every subsystem in which we collectively model the interconnection inputs and the event-triggering error as adversarial players that deteriorate the subsystem performance and model the control policy as the performance optimizer, competing against these adversarial players. To obtain a solution to this game, one has to solve the associated Hamilton-Jacobi-Isaac (HJI) equation, which does not have a closed-form solution even when the subsystem dynamics are accurately known. In this context, we introduce an event-driven off-policy integral reinforcement learning (OIRL) approach to learn an approximate solution to this HJI equation using artificial neural networks (NNs). We then use this NN approximated solution to design the control policy and event-triggering threshold at each subsystem. In the learning framework, we guarantee the Zeno-free behavior of the ETMs at each subsystem using the exploration policies. Finally, we derive sufficient conditions to guarantee uniform ultimate bounded regulation of the controlled system states and demonstrate the efficacy of the proposed framework with numerical examples.

Recommended Citation

V. Narayanan et al., "Event-Driven Off-Policy Reinforcement Learning for Control of Interconnected Systems," IEEE Transactions on Cybernetics, vol. 52, no. 3, pp. 1936 - 1946, Institute of Electrical and Electronics Engineers, Mar 2022.

The definitive version is available at https://doi.org/10.1109/TCYB.2020.2991166

Department(s)

Electrical and Computer Engineering

Comments

National Science Foundation, Grant CMMI 1547042

Keywords and Phrases

Decentralized control; differential game; event-triggered learning; off-policy learning

International Standard Serial Number (ISSN)

2168-2275; 2168-2267

Document Type

Article - Journal

Document Version

Final Version

File Type

text

Language(s)

English

Rights

© 2023 Institute of Electrical and Electronics Engineers, All rights reserved.

Publication Date

01 Mar 2022

PubMed ID

32639933