Multimodal Fusion of Voice and Gesture Data for UAV Control

To enable unmanned aerial vehicle (UAV) operators to efficiently and intuitively convey their commands to a swarm of UAVs, we propose the use of natural and human-centric input modalities, such as voices and gestures. This paper addresses the fusion of input modalities such as voice and gesture data, which are captured through a microphone and a Leap Motion controller, respectively, to control UAV swarms. The obtained experimental results are presented, and the achieved performance (accuracy) is analyzed. Finally, combined human factor ergonomics test with a questionnaire to verify the method's validity.

Automated summary

This paper proposes the use of natural and human-centric input modalities, such as voices and gestures, to control UAV swarms. It discusses the fusion of voice and gesture data captured through a microphone and a Leap Motion controller, and presents experimental results and analysis of the achieved performance (accuracy). Finally, it validates the method's effectiveness through combined human factor ergonomics test with a questionnaire.