Distributed response to network intrusions using multiagent reinforcement learning

Distributed denial of service (DDoS) attacks constitute a rapidly evolving threat in the current Internet. Multiagent Router Throttling is a novel approach to defend against DDoS attacks where multiple reinforcement learning agents are installed on a set of routers and learn to throttle or rate-limit traffic towards a victim server. It has been demonstrated to perform well against DDoS attacks in small-scale network topologies. The focus of this paper is to tackle the scalability challenge. Scalability is one of the most important aspects of a defence system since a non-scalable defence mechanism will never be considered, let alone adopted, for wide deployment by a company or organisation. In this paper we introduce Coordinated Team Learning (CTL) which is a novel design to the original Multiagent Router Throttling approach. One of the novel characteristics of our approach is that it provides a decentralised coordinated response to the DDoS problem. It incorporates several mechanisms, namely, hierarchical team-based communication, task decomposition and team rewards and its scalability is successfully demonstrated in experiments involving up to 100 reinforcement learning agents. We compare our proposed approach against a baseline and a popular state-of-the-art router throttling technique from the network security literature and we show that our approach significantly outperforms both of them in a series of scenarios with increasingly sophisticated attack dynamics. Furthermore, we show that our approach is more resilient and adaptable than the existing throttling approaches. (C) 2015 Elsevier Ltd. All rights reserved.