Facemap: a framework for modeling neural activity based on orofacial tracking

Recent studies in mice have shown that orofacial behaviors drive a large fraction of neural activity across the brain. To understand the nature and function of these signals, we need better computational models to characterize the behaviors and relate them to neural activity. Here we developed Facemap, a framework consisting of a keypoint tracker and a deep neural network encoder for predicting neural activity. Our algorithm for tracking mouse orofacial behaviors was more accurate than existing pose estimation tools, while the processing speed was several times faster, making it a powerful tool for real-time experimental interventions. The Facemap tracker was easy to adapt to data from new labs, requiring as few as 10 annotated frames for near-optimal performance. We used the keypoints as inputs to a deep neural network which predicts the activity of similar to 50,000 simultaneously-recorded neurons and, in visual cortex, we doubled the amount of explained variance compared to previous methods. Using this model, we found that the neuronal activity clusters that were well predicted from behavior were more spatially spread out across cortex. We also found that the deep behavioral features from the model had stereotypical, sequential dynamics that were not reversible in time. In summary, Facemap provides a stepping stone toward understanding the function of the brain-wide neural signals and their relation to behavior.

Automated summary

Recent studies in mice have found that orofacial behaviors have a significant impact on neural activity in the brain. To better understand these signals and their functions, researchers developed Facemap, a framework consisting of a keypoint tracker and a deep neural network encoder. The algorithm for tracking mouse orofacial behaviors was more accurate and faster than existing tools, making it useful for real-time experimental interventions. Using the deep neural network, they were able to predict the activity of thousands of neurons and found that neural activity clusters related to behavior were more spread out across the cortex. They also discovered that the behavioral features had sequential dynamics that were irreversible in time. Facemap provides a stepping stone towards understanding the relationship between neural signals and behavior.