Multi-agent hierarchical policy gradient for Air Combat Tactics emergence via self-play

Air-to-air confrontation has attracted wide attention from artificial intelligence scholars. However, in the complex air combat process, operational strategy selection depends heavily on aviation expert knowledge, which is usually expensive and difficult to obtain. Moreover, it is challenging to select optimal action sequences efficiently and accurately with existing methods, due to the high complexity of action selection when involving hybrid actions, e.g., discrete/continuous actions. In view of this, we propose a novel Multi-Agent Hierarchical Policy Gradient algorithm (MAHPG), which is capable of learning various strategies and transcending expert cognition by adversarial self-play learning. Besides, a hierarchical decision network is adopted to deal with the complicated and hybrid actions. It has a hierarchical decision-making ability similar to humankind, and thus, reduces the action ambiguity efficiently. Extensive experimental results demonstrate that the MAHPG outperforms the state-of-the-art air combat methods in terms of both defense and offense ability. Notably, it is discovered that the MAHPG has the ability of Air Combat Tactics Interplay Adaptation, and new operational strategies emerged that surpass the level of experts.

Automated summary

The researchers proposed a novel Multi-Agent Hierarchical Policy Gradient algorithm (MAHPG), capable of learning various strategies and surpassing expert cognition through adversarial self-play learning. The algorithm adopts a hierarchical decision network to handle complex and hybrid actions, similar to human decision-making ability, effectively reducing action ambiguity. Experimental results demonstrate that MAHPG excels in defense and offense ability compared to state-of-the-art air combat methods.