Intelligent Wearable Virtual Reality (VR) Gaming Controller for People with Motor Disabilities

Current interaction methods for VR/AR headsets need hand-held devices/joysticks, head-mounted cameras or external sensors to detect hand and body movements in order to generate actionable commands. It is not feasible for the people with motor or movement disabilities. In the meanwhile, the current assistive technologies need complicated, cumbersome, and expensive equipment, which are not user-friendly, not portable, and still require certain motor control capabilities from the user. Our approach aims to bridge this gap by developing a compact, non-obtrusive and ergonomic wearable device, to measure physiological signals associated with human eye/facial gestures, and therefore generate simple and intuitive commands to interact with the VR/AR headset. Our innovation uses machine learning and non-invasive biosensors attached on top of the ears (temple positions) to identify eye movements and facial expressions. Through proper design of Human-Machine interactions, we have come up with several engaging human-game multimodal interaction methods by simply moving user's eyes and doing facial expressions. It allows people with motor impairment have a fun time to play VR games with other people, totally hand-free.