Leveraging Currency for Repairing Inconsistent and Incomplete Data

Data quality plays a key role in big data management today. With the explosive growth of data from a variety of sources, the quality of data is faced with multiple problems. Motivated by this, we study the multiple data cleaning on incompleteness and inconsistency with currency reasoning and determination in this paper. We introduce a 4-step framework, named Imp3C, for errors detection and quality improvement in incomplete and inconsistent data without timestamps. We achieve an integrated currency determining method to compute the currency orders among tuples, according to currency constraints. Thus, the inconsistent data and missing values are repaired effectively considering the temporal impact. For both effectiveness and efficiency consideration, we carry out inconsistency repair ahead of incompleteness repair. A currency-related consistency distance metric is defined to measure the similarity between dirty tuples and clean ones more accurately. In addition, currency orders are treated as an important feature in the missing imputation training process. The solution algorithms are introduced in detail with case studies. A thorough experiment on three real-life datasets verifies our method Imp3C improves the performance of data repairing with multiple quality problems. Imp3C outperforms the existing advanced methods, especially in the datasets with complex currency orders.

Automated summary

This paper studies the method of multiple data cleaning for incomplete and inconsistent data, proposing a four-step framework called Imp3C. It uses a currency determining method and a consistency distance metric to repair dirty data. Experiments demonstrate that Imp3C outperforms existing methods in data repairing.