Model-Building Semi-Markov Adaptive Critics

Abstract

Adaptive or actor critics are a class of reinforcement learning (RL) or approximate dynamic programming (ADP) algorithms in which one searches over stochastic policies in order to determine the optimal deterministic policy. Classically, these algorithms have been studied for Markov decision processes (MDPs) in the context of model-free updates in which transition probabilities are avoided altogether. A model-free version for the semi-MDP (SMDP) for discounted reward in which the transition time of each transition can be a random variable was proposed in Gosavi [1]. In this paper, we propose a variant in which the transition probability model is built simultaneously with the value function and action-probability functions. While our new algorithm does not require the transition probabilities apriori, it generates them along with the estimation of the value function and the action-probability functions required in adaptive critics. Model-building and model-based versions of algorithms have numerous advantages in contrast to their model-free counterparts. In particular, they are more stable and may require less training. However the additional steps of building the model may require increased storage in the computer's memory. In addition to enumerating potential application areas for our algorithm, we will analyze the advantages and disadvantages of model building.

Meeting Name

IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (2011: Apr. 11-15, Paris, France)

Department(s)

Engineering Management and Systems Engineering

Second Department

Psychological Science

Keywords and Phrases

Actor critic; Adaptive critic; Approximate dynamic programming; Apriori; Discounted reward; Markov Decision Processes; Model free; Potential applications; Semi-Markov; Stochastic policy; Transition probabilities; Transition time; Value functions; Algorithms; Artificial intelligence; Dynamic programming; Markov processes; Random variables; Reinforcement learning; Stochastic models

International Standard Book Number (ISBN)

978-1424498888

Document Type

Article - Conference proceedings

Document Version

Citation

File Type

text

Language(s)

English

Rights

© 2011 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.

Publication Date

15 Apr 2011

Share

 
COinS