Contrastive Learning via Local Activity

Contrastive learning (CL) helps deep networks discriminate between positive and negative pairs in learning. As a powerful unsupervised pretraining method, CL has greatly reduced the performance gap with supervised training. However, current CL approaches mainly rely on sophisticated augmentations, a large number of negative pairs and chained gradient calculations, which are complex to use. To address these issues, in this paper, we propose the local activity contrast (LAC) algorithm, which is an unsupervised method based on two forward passes and locally defined loss to learn meaningful representations. The learning target of each layer is to minimize the activation value difference between two forward passes, effectively overcoming the limitations of applying CL above mentioned. We demonstrated that LAC could be a very useful pretraining method using reconstruction as the pretext task. Moreover, through pretraining with LAC, the networks exhibited competitive performance in various downstream tasks compared with other unsupervised learning methods.

Automated summary

In this paper, the authors propose a method called local activity contrast (LAC) for unsupervised pretraining in deep learning. LAC uses two forward passes and a locally defined loss function to learn meaningful representations, overcoming the complexity of current contrastive learning methods. The authors demonstrate that LAC can be a useful pretraining method, and it exhibits competitive performance in various downstream tasks compared to other unsupervised learning methods.