CNNs/ViTs-CNNs/ViTs: Mutual distillation for unsupervised domain adaptation

Unsupervised Domain Adaptation (UDA) is a popular machine learning technique to reduce the distribution discrepancy among domains. In previous UDA methods, only convolutional neural networks (CNNs) or vision transformers (ViTs) are used as the backbone. Therefore, these methods are subjected to inherent characteristics of single-CNNs or single-ViTs. In fact, CNNs are more biased towards local textures, while ViTs are inclined to learn the shape information of images. In this paper, we consider that the model combining the above two bias properties is closer to the human visual neural system and achieves more robust performance. A novel mutual distillation method for UDA without any assistant networks is proposed. Mutual distillation between two backbones with complementary properties (i.e. CNNs & ViTs) can promote each other, leading to better domain knowledge transfer. Additionally, traditional domain-mixup approaches can only mix limited cross-domain information through linear interpolation. To encourage more cross-domain information interaction between the two backbones and bridge the domain gap, we propose a patch-mixup method, building a train of mixed intermediate domains composed of augmented patches. Meanwhile, a cross-domain semantic alignment loss is proposed to align the semantic information of these domains. Extensive experiments show that our method achieves state-of-the-art results on several standard UDA benchmarks, such as Office-31, Office-Home, ImageCLEF-DA and VisDA-2017.(c) 2022 Elsevier Inc. All rights reserved.

Automated summary

This paper proposes a novel unsupervised domain adaptation method that improves the transfer of domain knowledge by mutual distillation between two models with complementary properties (i.e. CNNs and ViTs). It also introduces a patch-mixup method to encourage cross-domain information interaction and a cross-domain semantic alignment loss to align the semantic information of different domains. Extensive experiments demonstrate that this method achieves state-of-the-art results on various standard UDA benchmarks.