Real-Time Closed-Loop Feedback in Behavioral Time Scales Using DeepLabCut

Computer vision approaches have made significant inroads into offline tracking of behavior and estimating animal poses. In particular, because of their versatility, deep-learning approaches have been gaining attention in behavioral tracking without any markers. Here, we developed an approach using DeepLabCut for real-time estimation of movement. We trained a deep-neural network (DNN) offline with high-speed video data of a mouse whisking, then transferred the trained network to work with the same mouse, whisking in real-time. With this approach, we tracked the tips of three whiskers in an arc and converted positions into a TTL output within behavioral time scales, i.e., 10.5 ms. With this approach, it is possible to trigger output based on movement of individual whiskers, or on the distance between adjacent whiskers. Flexible closed-loop systems like the one we have deployed here can complement optogenetic approaches and can be used to directly manipulate the relationship between movement and neural activity.

Automated summary

Computer vision techniques, particularly deep-learning approaches, have been increasingly used for offline tracking behavior and estimating animal poses without markers. This research developed a method using DeepLabCut for real-time estimation of mouse movements, training a DNN offline with high-speed video data and transferring the network to work in real-time with the same mouse.By tracking whisker movements and converting them into output signals within behavioral time scales, this approach can be used to trigger outputs based on individual whisker movements or distances between adjacent whiskers, complementing optogenetic approaches for manipulating movement and neural activity relationships directly.