A machine learning framework for investigating data breaches based on semantic analysis of adversary's attack patterns in threat intelligence repositories

With the ever increasing cases of cyber data breaches, the manual process of sifting through tons of security logs to investigate cyber-attacks is error-prone and time-consuming. Signature-based deep search solutions only give accurate results if the threat artifacts are precisely provided. With the burgeoning variety of sophisticated cyber threats having common attack patterns and utilizing the same attack tools, a timely investigation is nearly impossible. There is a need to automate the threat analysis process by mapping adversary's Tactics, Techniques and Procedures (TTPs) to attack goals and detection mechanisms. In this paper, a novel machine learning based framework is proposed that identifies cyber threats based on observed attack patterns. The framework semantically relates threats and TTPs extracted from wellknown threat sources with associated detection mechanisms to form a semantic network. This network is then used to determine threat occurrences by forming probabilistic relationships between threats and TTPs. The framework is trained using a TTP taxonomy dataset and the performance is evaluated with threat artifacts reported in threat reports. The framework efficiently identifies attacks with 92% accuracy and low false positives even in the case of lost and spurious TTPs. The average detection time of a data breach incident is 0.15 s for a network trained with 133 TTPs from 45 threat families. (C) 2019 Elsevier B.V. All rights reserved.