An improved agglomerative hierarchical clustering anomaly detection method for scientific data

Anomaly detection tries to find out the data that disobeys the rule of majority data or expected patterns. The traditional hierarchical clustering algorithms have been adopted to detect anomaly, but have the disadvantages of low effectiveness and unstability. So we propose an improved agglomerative hierarchical clustering method for anomaly detection. It dynamically adjusts the optimum clustering number according to the self-defined criterion to save the trouble of manually picking clustering number, and determines the optimum clustering distance mode according to cophenetic correlation coefficient to reduce the procedure of manually testing the suitable distance mode in each iteration. The performances of proposed method are verified on tensile test, HTRU2 and credit card dataset. Compared with the traditional methods, our method possesses the most comprehensive performance (the highest F-measure with less iterations), which shows effectiveness of anomaly detection. And compared with the traditional methods (single, complete, average, and centroid mode), our method achieves the best performance on tensile test and HTRU2 dataset, showing stronger generalization. Compared with other methods (Decision + Gradient Boosted Tree, Decision Trees + Decision Stump, etc) on credit card dataset, our method obtains similar accuracy, and ranks in the top level in the aspect of sensitivity.

Automated summary

The study proposed an improved hierarchical clustering method for anomaly detection, aiming to improve the effectiveness and stability of anomaly detection. By dynamically adjusting the clustering number and selecting the optimum clustering distance mode, manual effort is reduced and better performance is achieved.