Variation Robust Cross-Modal Metric Learning for Caricature Recognition

In this paper, a variation robust cross-modal metric learning (VR-(CML)-L-2) method is proposed for caricature recognition. The goal of caricature recognition is to match a caricature with a photo. This recognition process needs to deal with all kind of variations including different modalities, facial appearance exaggerations, changes in viewpoint, expression, and illumination, etc. All these variations lead to severe misalignment between features of caricatures and photos. To deal with these problems, a specifically designed facial landmark based feature extraction scheme is proposed, where features of caricatures and photos are extracted using different feature extraction steps. At each facial landmark, features of photos are extracted with fixed viewing angle and scale, while features of caricatures are extracted with different scales and viewing angles. To measure the similarity of these features, multiple cross-modal metrics are learned at different facial landmarks in one optimization framework to guarantee global optimum. As the measured features are from two modalities (caricature and photo), cross-modal metric is used to remove modality variations. Pooling at distance level is used during metric optimization to further align the features of caricatures and photos. The introduced pooling step makes the learning method more robust to variations. Experimental results demonstrate the effectiveness of the proposed method on two caricature datasets with various variations.