Q-learning-based Optimal Tracking For Uncertain Linear Discrete-time Systems Via Multitask Adaptive Optimal Feedback Control

Abstract

An online Q-learning scheme is introduced to solve the discrete-time (DT) finite-horizon optimal tracking problem for linear systems using state and output feedback with imperfect system knowledge. To circumvent the reliance on system models, we develop an Adaptive Dynamic Programming (ADP) approach to estimate the optimal Q-function parameters based on real-time input and output data. In our approach, we decompose the explicitly time-varying Q-function parameters into a time-varying basis and time-invariant weight vector with state feedback initially. To relax the state measurements, the controller is significantly extended from the traditional state-feedback formulation to an output-feedback implementation. We integrate an adaptive observer which simultaneously estimates the state vector and unknown system dynamics using input and output measurements because most conventional observers require the system dynamics for good performance. The use of the adaptive observer allows the overall control system to achieve near-optimal performance, significantly outperforming model-based observers with identical initial model mismatch. Finally, the framework is extended to a multi-task control setting, designed to facilitate rapid adaptation across a collection of related systems with differing dynamics. Leveraging concepts from lifelong learning, the Q-function parameters are decomposed into a shared knowledge-base and task-specific parameter vectors. This sparse decomposition structure allows new tasks to be learned efficiently and quickly, benefiting from the knowledge accumulated across previously solved tasks. The resulting scheme minimizes the average cost across multiple systems, demonstrates beneficial knowledge transfer between tasks, and further relaxes the requirement for prior knowledge of the system's nominal behavior. Simulation results verify the theoretical claims.

Department(s)

Electrical and Computer Engineering

Second Department

Computer Science

Publication Status

Full Text Access

Comments

Office of Naval Research, Grant N00014-21-1-2232

Keywords and Phrases

Finite-horizon optimal control; Online learning; Q-learning; Trajectory tracking

International Standard Serial Number (ISSN)

0952-1976

Document Type

Article - Journal

Document Version

Citation

File Type

text

Language(s)

English

Rights

© 2026 Elsevier; International Federation of Automatic Control (IFAC), All rights reserved.

Publication Date

15 Nov 2026

Link to Full Text

Share

 
COinS