Intelligent VNF Orchestration and Flow Scheduling via Model-Assisted Deep Reinforcement Learning

Hosting virtualized network functions (VNF) has been regarded as an effective way to realize network function virtualization (NFV). Considering the cost diversity in cloud computing, from the perspective of service providers, it is significant to orchestrate the VNFs and schedule the traffic flows for network utility maximization (NUM) as it implies maximal revenue. However, traditional heuristic solutions based on optimization models usually follow some assumptions, limiting their applicability. Recent studies have shown that deep reinforcement learning (DRL) is a promising way to tackle such limitations. However, DRL agent training also suffers from slow convergence problem, especially with complex control problems. We notice that optimization models actually can be applied to accelerate the DRL training. Therefore, we are motivated to design a model-assisted DRL framework for VNF orchestration in this paper. Other than letting the agent blindly explore actions, the heuristic solutions are used to guide the training process. Based on such principle, the DRL framework is also redesigned accordingly. Experiment results validate the high efficiency of our model-assisted DRL framework as it not only converges $23\times$ faster than traditional DRL algorithm, but also with higher performance at the same time.