Journal
ACM TRANSACTIONS ON KNOWLEDGE DISCOVERY FROM DATA
Volume 16, Issue 6, Pages -Publisher
ASSOC COMPUTING MACHINERY
DOI: 10.1145/3522592
Keywords
Data pre-processing scheme; influence space; noise identification; data representation; ranking factor
Funding
- National Natural Science Foundation of China [U1931209]
- Key Research and Development Projects of Shanxi Province [201903D121116, 20201070]
- Fundamental Research Program of Shanxi Province [20210302123223]
Ask authors/readers for more resources
This article proposes a two-stage data pre-processing framework called ARIS for noise identification and data reduction. The framework improves the accuracy of data analysis by identifying and removing noise, and reduces the impact of noise on the results.
The extensive growth of data quantity has posed many challenges to data analysis and retrieval. Noise and redundancy are typical representatives of the above-mentioned challenges, which may reduce the reliability of analysis and retrieval results and increase storage and computing overhead. To solve the above problems, a two-stage data pre-processing framework for noise identification and data reduction, called ARIS, is proposed in this article. The first stage identifies and removes noises by the following steps: First, the influence space (IS) is introduced to elaborate data distribution. Second, a ranking factor (RF) is defined to describe the possibility that the points are regarded as noises, then, the definition of noise is given based on RF. Third, a clean dataset (CD) is obtained by removing noise from the original dataset. The second stage learns representative data and realizes data reduction. In this process, CD is divided into multiple small regions by IS. Then the reduced dataset is formed by collecting the representations of each region. The performance of ARIS is verified by experiments on artificial and real datasets. Experimental results show that ARIS effectively weakens the impact of noise and reduces the amount of data and significantly improves the accuracy of data analysis within a reasonable time cost range.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available