Distance Estimation in Numerical Data Sets with Missing Values

Abstract

The Possibility of Missing or Incomplete Data is Often Ignored When Describing Statistical or Machine Learning Methods, But as It is a Common Problem in Practice, It is Relevant to Consider. a Popular Strategy is to Fill in the Missing Values by Imputation as a Pre-Processing Step, But for Many Methods This is Not Necessary, and Can Yield Sub-Optimal Results. Instead, Appropriately Estimating Pairwise Distances in a Data Set Directly Enables the Use of Any Machine Learning Methods using Nearest Neighbours or Otherwise based on Distances between Samples. in This Paper, It is Shown How Directly Estimating Distances Tends to Result in More Accurate Results Than Calculating Distances from an Imputed Data Set, and an Algorithm to Calculate the Estimated Distances is Presented. the Theoretical Framework Operates under the Assumption of a Multivariate Normal Distribution, But the Algorithm is Shown to Be Robust to Violations of This Assumption. the Focus is on Numerical Data with a Considerable Proportion of Missing Values, and Simulated Experiments Are Provided to Show Accurate Performance on Several Data Sets. © 2013 Elsevier Inc. All Rights Reserved.

Department(s)

Engineering Management and Systems Engineering

Keywords and Phrases

Distance estimation; Imputation; Missing data; Nearest neighbour

International Standard Serial Number (ISSN)

0020-0255

Document Type

Article - Journal

Document Version

Citation

File Type

text

Language(s)

English

Rights

© 2024 Elsevier, All rights reserved.

Publication Date

10 Aug 2013

Link to Full Text

Share

 
COinS