Model-Building Semi-Markov Adaptive Critics


Adaptive or actor critics are a class of reinforcement learning (RL) or approximate dynamic programming (ADP) algorithms in which one searches over stochastic policies in order to determine the optimal deterministic policy. Classically, these algorithms have been studied for Markov decision processes (MDPs) in the context of model-free updates in which transition probabilities are avoided altogether. A model-free version for the semi-MDP (SMDP) for discounted reward in which the transition time of each transition can be a random variable was proposed in Gosavi [1]. In this paper, we propose a variant in which the transition probability model is built simultaneously with the value function and action-probability functions. While our new algorithm does not require the transition probabilities apriori, it generates them along with the estimation of the value function and the action-probability functions required in adaptive critics. Model-building and model-based versions of algorithms have numerous advantages in contrast to their model-free counterparts. In particular, they are more stable and may require less training. However the additional steps of building the model may require increased storage in the computer's memory. In addition to enumerating potential application areas for our algorithm, we will analyze the advantages and disadvantages of model building.

Meeting Name

IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (2011: April 11-15; Paris, France)


Engineering Management and Systems Engineering

Second Department

Psychological Science


IEEE Computational Intelligence Society

Keywords and Phrases

Actor critic; Adaptive critic; Approximate dynamic programming; Apriori; Discounted reward; Markov Decision Processes; Model free; Potential applications; Semi-Markov; Stochastic policy; Transition probabilities; Transition time; Value functions; Algorithms; Artificial intelligence; Dynamic programming; Markov processes; Random variables; Reinforcement learning; Stochastic models

International Standard Book Number (ISBN)


Document Type

Article - Conference proceedings

Document Version


File Type





© 2011 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.