4.7 Article

Impact of benign sample size on binary classification accuracy

Journal

EXPERT SYSTEMS WITH APPLICATIONS
Volume 211, Issue -, Pages -

Publisher

PERGAMON-ELSEVIER SCIENCE LTD
DOI: 10.1016/j.eswa.2022.118630

Keywords

Malware; Machine learning; Binary classification; Benign sample; Random forest; Support vector machine; XGBoost

Ask authors/readers for more resources

There has been a significant increase in malware attacks and malicious traffic. Various machine learning-based detection models have been developed, but their evaluation methods and datasets differ, making it difficult to compare their performances accurately. This study proposes a new metric for evaluating accuracy degradation caused by increasing the benign sample size in binary classification. Using the FFRI dataset, the classification accuracy was evaluated with extracted strings from malware, and it was found that increasing the benign sample size resulted in a decrease in the F1 score.
Recently, there has been a significant increase in malware attacks and malicious traffic. Consequently, several machine learning-based detection models have been developed to detect them. However, the detection accuracy of these models is currently evaluated using different methodologies and datasets, with some studies overstating high detection rates. The lack of a common testing approach coupled with the limited datasets used for the experiments make it challenging to compare the performances of these models to identify those that provide superior detection accuracy. A few studies have focused on benign samples and their effects on detection accuracy. The datasets used in the experiments generally consist of benign and malicious samples; hence, binary classification is used in the machine learning models. In the binary classification task, the size of a benign sample affects the classification accuracy of malicious samples, that is, it can either improve or degrade detection accuracy. In this study, we propose a novel metric for evaluating accuracy degradation by increasing benign sample size. We mainly used the FFRI dataset, which consists of 11,243 malware samples and 250,000 benign samples, and evaluated the classification accuracy with extracted strings from the malware. In addition, we obtained other malware samples that we used as supplementary to the main dataset. We increased the number of benign samples for testing by tenfold, while maintaining the malicious sample and benign training sample sizes, which resulted in a decrease of 0.293 in the F1 score. Furthermore, we confirmed that using a sufficiently sized benign training sample set mitigates accuracy degradation. Our metric can be beneficial for evaluating the benign sample size needed in binary classification and comparing accuracy.

Authors

I am an author on this paper
Click your name to claim this paper and add it to your profile.

Reviews

Primary Rating

4.7
Not enough ratings

Secondary Ratings

Novelty
-
Significance
-
Scientific rigor
-
Rate this paper

Recommended

No Data Available
No Data Available