Model-Building Semi-Markov Adaptive Critics
Adaptive or actor critics are a class of reinforcement learning (RL) or approximate dynamic programming (ADP) algorithms in which one searches over stochastic policies in order to determine the optimal deterministic policy. Classically, these algorithms have been studied for Markov decision processes (MDPs) in the context of model-free updates in which transition probabilities are avoided altogether. A model-free version for the semi-MDP (SMDP) for discounted reward in which the transition time of each transition can be a random variable was proposed in Gosavi . In this paper, we propose a variant in which the transition probability model is built simultaneously with the value function and action-probability functions. While our new algorithm does not require the transition probabilities apriori, it generates them along with the estimation of the value function and the action-probability functions required in adaptive critics. Model-building and model-based versions of algorithms have numerous advantages in contrast to their model-free counterparts. In particular, they are more stable and may require less training. However the additional steps of building the model may require increased storage in the computer's memory. In addition to enumerating potential application areas for our algorithm, we will analyze the advantages and disadvantages of model building.
A. Gosavi et al., "Model-Building Semi-Markov Adaptive Critics," Proceedings of the IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (2011,; Paris, France), pp. 170-175, Institute of Electrical and Electronics Engineers (IEEE), Jan 2011.
The definitive version is available at https://doi.org/10.1109/ADPRL.2011.5967374
IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (2011: April 11-15; Paris, France)
Engineering Management and Systems Engineering
IEEE Computational Intelligence Society
Keywords and Phrases
Actor critic; Adaptive critic; Approximate dynamic programming; Apriori; Discounted reward; Markov Decision Processes; Model free; Potential applications; Semi-Markov; Stochastic policy; Transition probabilities; Transition time; Value functions; Algorithms; Artificial intelligence; Dynamic programming; Markov processes; Random variables; Reinforcement learning; Stochastic models
International Standard Book Number (ISBN)
Article - Conference proceedings
© 2011 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.