Redundant or duplicate data are the most troublesome problem in database management and applications. Approximate field matching is the key solution to resolve the problem by identifying semantically equivalent string values in syntactically different representations. This paper considers token-based solutions and proposes a general field matching framework to generalize the field matching problem in different domains. By introducing a concept of String Matching Points (SMP) in string comparison, string matching accuracy and efficiency are improved, compared with other commonly-applied field matching algorithms. The paper discusses the development of field matching algorithms from the developed general framework. The framework and corresponding algorithm are tested on a public data set of the NASA publication abstract database. The approach can be applied to address the similar problems in other databases.
M. Wei et al., "Improving Database Quality through Eliminating Duplicate Records," Data Science Journal, vol. 5, pp. 127-142, Committee on Data for Science and Technology, Nov 2006.
The definitive version is available at http://dx.doi.org/10.2481/dsj.5.127
Geosciences and Geological and Petroleum Engineering
Keywords and Phrases
Field Matching; General Field Matching Framework; String Matching Patterns; String Matching Points; Algorithms; Electronic Publishing; Problem Solving; Records Management; Semantics; Database Systems
International Standard Serial Number (ISSN)
Article - Journal
© 2006 Committee on Data for Science and Technology, All rights reserved.