Unsupervised deep learning identifies semantic disentanglement in single inferotemporal face patch neurons

Little is known about the brain's computations that enable the recognition of faces. Here, the authors use unsupervised deep learning to show that the brain disentangles faces into semantically meaningful factors, like age or the presence of a smile, at the single neuron level. In order to better understand how the brain perceives faces, it is important to know what objective drives learning in the ventral visual stream. To answer this question, we model neural responses to faces in the macaque inferotemporal (IT) cortex with a deep self-supervised generative model, beta-VAE, which disentangles sensory data into interpretable latent factors, such as gender or age. Our results demonstrate a strong correspondence between the generative factors discovered by beta-VAE and those coded by single IT neurons, beyond that found for the baselines, including the handcrafted state-of-the-art model of face perception, the Active Appearance Model, and deep classifiers. Moreover, beta-VAE is able to reconstruct novel face images using signals from just a handful of cells. Together our results imply that optimising the disentangling objective leads to representations that closely resemble those in the IT at the single unit level. This points at disentangling as a plausible learning objective for the visual brain.

Automated summary

The study explores how the brain processes facial recognition by using deep learning techniques, revealing that the brain disentangles facial images into semantically meaningful factors such as age or smile. Through a deep self-supervised generative model, beta-VAE, the researchers model neural responses in the macaque IT cortex to faces, demonstrating a strong correspondence between generative factors and single IT neurons. This suggests that optimizing disentangling objectives may lead to representations that closely resemble those in the IT at the single unit level.