Online Concurrent Reinforcement Learning Algorithm to Solve Two-Player Zero-Sum Games for Partially Unknown Nonlinear Continuous-Time Systems

Abstract

Online adaptive optimal control methods based on reinforcement learning algorithms typically need to check for the persistence of excitation condition, which is necessary to be known a priori for convergence of the algorithm. However, this condition is often infeasible to implement or monitor online. This paper proposes an online concurrent reinforcement learning algorithm (CRLA) based on neural networks (NNs) to solve the H∞ control problem of partially unknown continuous-time systems, in which the need for persistence of excitation condition is relaxed by using the idea of concurrent learning. First, H∞ control problem is formulated as a two-player zero-sum game, and then, online CRLA is employed to obtain the approximation of the optimal value and the Nash equilibrium of the game. The proposed algorithm is implemented on actor-critic-disturbance NN approximator structure to obtain the solution of the Hamilton-Jacobi-Isaacs equation online forward in time. During the implementation of the algorithm, the control input that acts as one player attempts to make the optimal control while the other player, that is, disturbance, tries to make the worst-case possible disturbance. Novel update laws are derived for adaptation of the critic and actor NN weights. The stability of the closed-loop system is guaranteed using Lyapunov technique, and the convergence to the Nash solution of the game is obtained. Simulation results show the effectiveness of the proposed method.

Department(s)

Electrical and Computer Engineering

Keywords and Phrases

Algorithms; Closed Loop Systems; Concurrency Control; E-Learning; Game Theory; Learning Algorithms; Neural Networks; Nonlinear Control Systems; Online Systems; Optimal Control Systems; Reinforcement Learning; Social Networking (online); Telecommunication Networks; Adaptive Optimal Control; Hamilton-Jacobi-Isaacs Equations; Lyapunov Techniques; Neural Networks (NNS); Nonlinear Continuous-Time Systems; Optimal Controls; Persistence of Excitation; Zero-Sum Game; Continuous Time Systems; Control; Online Concurrent Reinforcement Learning Algorithm; Two-Player Zero-Sum Games

International Standard Serial Number (ISSN)

0890-6327

Document Type

Article - Journal

Document Version

Citation

File Type

text

Language(s)

English

Rights

© 2015 John Wiley & Sons, All rights reserved.

Publication Date

01 Apr 2015

Link to Full Text

Share

 
COinS