Outlier detection using an ensemble of clustering algorithms

Outlier detection is an important research area in the field of machine learning and data science. The presence of outliers in a dataset limits its true usefulness in a real-life scenario. Due to the varied challenges, researchers strive to find a general method to be useful for different datasets. In this paper, we have proposed an outlier detection technique based on unsupervised learning using an ensemble of three clustering algorithms, namely K-means, K-means++ and Fuzzy C-means. We have proposed a unique way to deal with clustered outliers. Outcomes of the three aforementioned clustering algorithms are combined intelligently to accumulate all the complementary information. To combine the decisions of the hard and soft clustering algorithms, we have proposed a novel probability-based technique, which assigns a membership value to each data point in the case of a hard clustering algorithm. Three cluster validity indices are used as our evaluation metrics, which measure the goodness of a cluster. Significant improvement of cluster validity indices is observed after removing the outliers, which ensures the removal of outliers has resulted in stringent clusters. The method is evaluated on eight datasets, among which, three datasets are comparatively large. Source code of this work is available at: https://github.com/biswarup9/Outlier-Detection-Using-an-Ensemble-of-Clustering-Algorithms-.

Automated summary

Outlier detection is vital in machine learning and data science, with this research proposing a technique using an ensemble of three clustering algorithms and novel probability-based methods for handling clustered outliers. Significant improvement in cluster validity metrics was observed after removing the outliers.