Brain-inspired multimodal hybrid neural network for robot place recognition

Place recognition is an essential spatial intelligence capability for robots to understand and navigate the world. However, recognizing places in natural environments remains a challenging task for robots because of resource limitations and changing environments. In contrast, humans and animals can robustly and efficiently recognize hundreds of thousands of places in different conditions. Here, we report a brain-inspired general place recog-nition system, dubbed NeuroGPR, that enables robots to recognize places by mimicking the neural mechanism of multimodal sensing, encoding, and computing through a continuum of space and time. Our system consists of a multimodal hybrid neural network (MHNN) that encodes and integrates multimodal cues from both con-ventional and neuromorphic sensors. Specifically, to encode different sensory cues, we built various neural net-works of spatial view cells, place cells, head direction cells, and time cells. To integrate these cues, we designed a multiscale liquid state machine that can process and fuse multimodal information effectively and asynchronous-ly using diverse neuronal dynamics and bioinspired inhibitory circuits. We deployed the MHNN on Tianjic, a hybrid neuromorphic chip, and integrated it into a quadruped robot. Our results show that NeuroGPR achieves better performance compared with conventional and existing biologically inspired approaches, exhibiting ro-bustness to diverse environmental uncertainty, including perceptual aliasing, motion blur, light, or weather changes. Running NeuroGPR as an overall multi-neural network workload on Tianjic showcases its advantages with 10.5 times lower latency and 43.6% lower power consumption than the commonly used mobile robot pro-cessor Jetson Xavier NX.

Automated summary

The research report introduces a brain-inspired general place recognition system called NeuroGPR, which enables robots to recognize places in natural environments by mimicking the neural mechanism of multimodal sensing, encoding, and computing. The system utilizes a multimodal hybrid neural network to encode and integrate cues from different sensors, and a multiscale liquid state machine to process and fuse the information. Experimental results show that NeuroGPR performs well in various environmental conditions.