Model-Building Semi-Markov Adaptive Critics
Abstract
Adaptive or actor critics are a class of reinforcement learning (RL) or approximate dynamic programming (ADP) algorithms in which one searches over stochastic policies in order to determine the optimal deterministic policy. Classically, these algorithms have been studied for Markov decision processes (MDPs) in the context of model-free updates in which transition probabilities are avoided altogether. A model-free version for the semi-MDP (SMDP) for discounted reward in which the transition time of each transition can be a random variable was proposed in Gosavi [1]. In this paper, we propose a variant in which the transition probability model is built simultaneously with the value function and action-probability functions. While our new algorithm does not require the transition probabilities apriori, it generates them along with the estimation of the value function and the action-probability functions required in adaptive critics. Model-building and model-based versions of algorithms have numerous advantages in contrast to their model-free counterparts. In particular, they are more stable and may require less training. However the additional steps of building the model may require increased storage in the computer's memory. In addition to enumerating potential application areas for our algorithm, we will analyze the advantages and disadvantages of model building.
Recommended Citation
A. Gosavi et al., "Model-Building Semi-Markov Adaptive Critics," Proceedings of the IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (2011, Paris, France), pp. 170 - 175, Institute of Electrical and Electronics Engineers (IEEE), Apr 2011.
The definitive version is available at https://doi.org/10.1109/ADPRL.2011.5967374
Meeting Name
IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (2011: Apr. 11-15, Paris, France)
Department(s)
Engineering Management and Systems Engineering
Second Department
Psychological Science
Keywords and Phrases
Actor critic; Adaptive critic; Approximate dynamic programming; Apriori; Discounted reward; Markov Decision Processes; Model free; Potential applications; Semi-Markov; Stochastic policy; Transition probabilities; Transition time; Value functions; Algorithms; Artificial intelligence; Dynamic programming; Markov processes; Random variables; Reinforcement learning; Stochastic models
International Standard Book Number (ISBN)
978-1424498888
Document Type
Article - Conference proceedings
Document Version
Citation
File Type
text
Language(s)
English
Rights
© 2011 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.
Publication Date
15 Apr 2011
