Varying sex and identity of faces affects face categorization differently in humans and computational models

Our faces display socially important sex and identity information. How perceptually independent are these facial characteristics? Here, we used a sex categorization task to investigate how changing faces in terms of either their sex or identity affects sex categorization of those faces, whether these manipulations affect sex categorization similarly when the original faces were personally familiar or unknown, and, whether computational models trained for sex classification respond similarly to human observers. Our results show that varying faces along either sex or identity dimension affects their sex categorization. When the sex was swapped (e.g., female faces became male looking, Experiment 1), sex categorization performance was different from that with the original unchanged faces, and significantly more so for people who were familiar with the original faces than those who were not. When the identity of the faces was manipulated by caricaturing or anti-caricaturing them (these manipulations either augment or diminish idiosyncratic facial information, Experiment 2), sex categorization performance to caricatured, original, and anti-caricatured faces increased in that order, independently of face familiarity. Moreover, our face manipulations showed different effects upon computational models trained for sex classification and elicited different patterns of responses in humans and computational models. These results not only support the notion that the sex and identity of faces are processed integratively by human observers but also demonstrate that computational models of face categorization may not capture key characteristics of human face categorization.

Automated summary

Our faces convey important social information about sex and identity, and these characteristics are perceptually independent. Manipulating faces in terms of sex or identity affects sex categorization, with greater impact for familiar faces. Caricaturing or anti-caricaturing facial identity also influences sex categorization, regardless of familiarity, showing an increase in performance from caricatured to original to anti-caricatured faces. Computational models trained for sex classification respond differently and exhibit distinct patterns of response compared to human observers.