Abstract
Choosing the appropriate forecasting technique to employ is a challenging issue and requires a comprehensive analysis of empirical results. Recent research findings reveal that the performance evaluation of forecasting models depends on the accuracy measures adopted. Some methods indicate superior performance when error based metrics are used, while others perform better when precision values are adopted as accuracy measures. As scholars tend to use a smaller subset of accuracy metrics to assess the performance of forecasting models, there is a need for a concept of multiple accuracy dimensions to assure the robustness of evaluation. Therefore, the main purpose of this paper is to propose a decision making model that allows researchers to identify the superiority of a forecasting technique over another by considering several accuracy metrics concurrently. A multi-criteria decision analysis approach, namely the preference ranking organization method for enrichment evaluation (PROMETHEE), was adopted to solve this problem. Bayesian Networks, Artificial Neural Networks, SVMs, Logistic Regression, and several Rule and Tree-based forecasting approaches were included in the analysis. After introducing a detailed description of accuracy measures, the performance of the prediction models are evaluated using a chosen dataset from the UCI Machine Learning Repository.
Recommended Citation
N. Mehdiyev et al., "Evaluating Forecasting Methods by Considering Different Accuracy Measures," Procedia Computer Science, vol. 95, pp. 264 - 271, Elsevier, Nov 2016.
The definitive version is available at https://doi.org/10.1016/j.procs.2016.09.332
Meeting Name
Complex Adaptive Systems (2016: Nov. 2-4, Los Angeles, CA)
Department(s)
Engineering Management and Systems Engineering
Research Center/Lab(s)
Intelligent Systems Center
Keywords and Phrases
Adaptive systems; Artificial intelligence; Bayesian networks; Classification (of information); Complex networks; Decision making; Learning systems; Neural networks; Accuracy measures; Comprehensive analysis; Confusion matrices; Decision making models; MCDA; Multi-criteria decision analysis; PROMETHEE; UCI machine learning repository; Forecasting; Confusion Matrix
International Standard Serial Number (ISSN)
1877-0509
Document Type
Article - Conference proceedings
Document Version
Final Version
File Type
text
Language(s)
English
Rights
© 2016 The Authors, All rights reserved.
Creative Commons Licensing
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 License.
Publication Date
01 Nov 2016
Comments
This research was funded in part by the German Federal Ministry of Education and Research under grant number 01IS12050 (project PRODIGY) and 01IS14004A (project iPRODICT).