Leveraging malicious behavior traces from volatile memory using machine learning methods for trusted unknown malware detection in Linux cloud environments

Most organizations today use cloud-computing environments and virtualization technology. Linux-based clouds are the most popular cloud environments among organizations, and thus have become the target of cyber-attacks launched by sophisticated malware. Existing malware detection solutions for Linux-based VMs are installed and operated on the VM itself and are considered untrusted since malware can detect, interfere with, and even evade them. Thus, Linux cloud-based environments remain exposed to various malware-based attacks. This paper presents the first trusted framework for detecting unknown malware in Linux VM cloud-environments. Our framework acquires volatile memory dumps from the inspected VM by querying the hypervisor in a trusted manner and overcoming malware's ability to detect the security mechanism and evade detection. Then, using machine-learning algorithms we leverage informative traces (our 171 proposed features) from different parts of the VM's volatile memory. The framework was evaluated in seven rigorous experiments, on a total of 21,800 volatile memory dumps taken from two widely used virtual servers (10,900 from each server) during the execution of a diverse yet representative collection of benign and malicious Linux applications. Notably, the results show that our proposed framework can accurately (with high TPRs and low FPRs): (a) detect unknown malware (b) detect new unknown malware from unseen malware categories, which is a critical ability for coping with new malware trends and phenomena; (c) categorize an unknown malware by its attack category; (d) detect unknown malware on an unknown virtual-server; and lastly (e) detect fileless malware, a critical capability demonstrating the ability to detect substantially different attack modus operandi. (C) 2021 Elsevier B.V. All rights reserved.

Automated summary

This paper presents the first trusted framework for detecting unknown malware in Linux VM cloud-environments using machine-learning algorithms and informative traces from volatile memory. The framework was rigorously evaluated in experiments and showed high accuracy in detecting unknown malware and categorizing them by attack category.