Malware classification using dynamic features and Hidden Markov Model

In recent years the number of new malware threats has increased significantly, causing a damage of billions of dollars globally. To counter this aggressive malware attack, the anti-malware industry needs to be able to correctly classify malware in order to provide defense against them. Consequently, malware classification has been an active area of research, and a multitude of malware classification approaches have been proposed in the literature. This paper evaluates two methods of sequence classification based on Hidden Markov Model, namely the maximum likelihood and similarity-based methods, for classification of malware using a large and comprehensive dataset. System calls generated by known malware during execution are used as observation sequences to train the Hidden Markov Models. Malware samples are evaluated against the trained models to produce similarity vectors, which are used in the maximum likelihood and similarity-based classification schemes to predict the family for an unknown malware sample. Comparison of the two schemes shows that combining the powerful statistical pattern analysis capability of Hidden Markov Models and discriminative classifiers in the similarity based method results in a significantly better classification performance as compared to the maximum likelihood approach. Furthermore, evaluation of different classifiers in the similarity-based method demonstrates that Random Forest classifier performs better than other classifiers on malware similarity vectors.