Adaptive Generative Initialization in Transfer Learning

In spite of numerous researches on transfer learning, the consensus on the optimal method in transfer learning has not been reached. To render a unified theoretical understanding of transfer learning, we rephrase the crux of transfer learning as pursuing the optimal initialisation in facilitating the to-be-transferred task. Hence, to obtain an ideal initialisation, we propose a novel initialisation technique, i.e., adapted generative initialisation. Not limit to boost the task transfer, more importantly, the proposed initialisation can also bound the transfer benefits in defending the devastating negative transfer. At first stage in our proposed initialisation, the in-congruency between a task and its assigned learner (model) can be alleviated through feeding the knowledge of the target learner to train the source learner, whereas the later generative stage ensures the adapted initialisation can be properly produced to the target learner. The superiority of our proposed initialisation over conventional neural network based approaches was validated in our preliminary experiment on MNIST dataset.