Semi-supervised Cycle-GAN for face photo-sketch translation in the wild

The performance of face photo-sketch translation has improved a lot thanks to deep neural networks. GAN based methods trained on paired images can produce high-quality results under laboratory settings. Such paired datasets are, however, often very small and lack diversity. Meanwhile, Cycle-GANs trained with unpaired photo-sketch datasets suffer from the steganography phenomenon, which makes them not effective to face photos in the wild. In this paper, we introduce a semi-supervised approach with a noise-injection strategy, named Semi-Cycle-GAN (SCG), to tackle these problems. For the first problem, we propose a pseudo sketch feature representation for each input photo composed from a small reference set of photo-sketch pairs, and use the resulting pseudo pairs to supervise a photo-to-sketch generator ������������2 ������. The outputs of ������������2 ������ can in turn help to train a sketch-to-photo generator ������������2 ������ in a self-supervised manner. This allows us to train ������������2 ������ and ������������2 ������ using a small reference set of photo-sketch pairs together with a large face photo dataset (without ground-truth sketches). For the second problem, we show that the simple noise-injection strategy works well to alleviate the steganography effect in SCG and helps to produce more reasonable sketch-to-photo results with less overfitting than fully supervised approaches. Experiments show that SCG achieves competitive performance on public benchmarks and superior results on photos in the wild.

Automated summary

This paper proposes a semi-supervised approach, named Semi-Cycle-GAN (SCG), to tackle the problems of face photo-sketch translation. The approach utilizes a noise-injection strategy and pseudo sketch feature representation to generate high-quality results. It also alleviates the steganography effect and overfitting issues commonly seen in fully supervised approaches.