Journal
NEUROSCIENCE
Volume 444, Issue -, Pages 92-105Publisher
PERGAMON-ELSEVIER SCIENCE LTD
DOI: 10.1016/j.neuroscience.2020.07.040
Keywords
brain decoding; visual reconstruction; generative adversarial network (GAN); Bayesian framework
Categories
Funding
- National Key Research and Development Plan of China [2017YFB1002502]
- National Natural Science Foundation of China [61701089]
- Natural Science Foundation of Henan Province of China [162300410333]
Ask authors/readers for more resources
In the visual decoding domain, the most difficult task is the visual reconstruction aimed at reconstructing the presented visual stimuli given the corresponding human brain activity monitored by functional magnetic resonance imaging (fMRI), especially when reconstructing viewed natural images. Recent research regarded the visual reconstruction as the conditional image generation on fMRI voxels and started to use the generative adversarial networks (GANs) to design computational models for this task. Despite the great improvement in previous GAN-based methods, the fidelity and naturalness of the reconstructed images are still unsatisfactory, the reasons include the small number of fMRI data samples and the instability of GAN training. In this study, we propose a new GAN-based Bayesian visual reconstruction model (GAN-BVRM) to avoid the contradiction between naturalness and fidelity in current GAN-based methods. GAN-BVRM is composed of a classifier to decode the categories from fMRI data, a pre-trained conditional generator of the distinguished BigGAN to generate natural images of the specified categories, and a set of encoding models and an evaluator to evaluate the generated images. Composed of neural networks, GAN-BVRM is fully differentiable and can directly generate the reconstructed images by iteratively updating the noise input vector through backpropagation to fit the fMRI voxels. In this process, the decoded categories and encoding models are responsible for the semantic and detailed contents of the reconstructed images, respectively. Experimental results revealed that GAN-BVRM improved the fidelity and naturalness, which validated the advantage of the combining of GANs and Bayesian manner for visual reconstruction. (C) 2020 IBRO. Published by Elsevier Ltd. All rights reserved.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available