REMIX: Automated Exploration for Interactive Outlier Detection
Outlier detection is the identification of points in a dataset that do not conform to the norm. Outlier detection is highly sensitive to the choice of the detection algorithm and the feature subspace used by the algorithm. Extracting domain-relevant insights from outliers needs systematic exploration of these choices since diverse outlier sets could lead to complementary insights. This challenge is especially acute in an interactive setting, where the choices must be explored in a time-constrained manner.
In this work, we present REMIX, the first system to address the problem of outlier detection in an interactive setting. REMIX uses a novel mixed integer programming (MIP) formulation for automatically selecting and executing a diverse set of outlier detectors within a time limit. This formulation incorporates multiple aspects such as (i) an upper limit on the total execution time of detectors (ii) diversity in the space of algorithms and features, and (iii) meta-learning for evaluating the cost and utility of detectors. REMIX provides two distinct ways for the analyst to consume its results: (i) a partitioning of the detectors explored by REMIX into perspectives through low-rank non-negative matrix factorization; each perspective can be easily visualized as an intuitive heatmap of experiments versus outliers, and (ii) an ensembled set of outliers which combines outlier scores from all detectors. We demonstrate the benefits of REMIX through extensive empirical validation on real-world data.
Y. Fu et al., "REMIX: Automated Exploration for Interactive Outlier Detection," Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2017, Halifax, Canada), pp. 827 - 835, Association for Computing Machinery (ACM), Sep 2017.
The definitive version is available at https://doi.org/10.1145/3097983.3098154
23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2017 (2017: Aug. 13-17, Halifax, Canada)
Keywords and Phrases
Data handling; Data mining; Factorization; Integer programming; Statistics; Detection algorithm; Empirical validation; Feature subspace; First systems; Mixed integer programming (MIP); Nonnegative matrix factorization; Outlier Detection; Systematic exploration; Feature extraction
International Standard Book Number (ISBN)
Article - Conference proceedings
© 2017 Association for Computing Machinery (ACM), All rights reserved.
01 Sep 2017