期刊
NEUROSCIENCE
卷 444, 期 -, 页码 92-105出版社
PERGAMON-ELSEVIER SCIENCE LTD
DOI: 10.1016/j.neuroscience.2020.07.040
关键词
brain decoding; visual reconstruction; generative adversarial network (GAN); Bayesian framework
资金
- National Key Research and Development Plan of China [2017YFB1002502]
- National Natural Science Foundation of China [61701089]
- Natural Science Foundation of Henan Province of China [162300410333]
In the visual decoding domain, the most difficult task is the visual reconstruction aimed at reconstructing the presented visual stimuli given the corresponding human brain activity monitored by functional magnetic resonance imaging (fMRI), especially when reconstructing viewed natural images. Recent research regarded the visual reconstruction as the conditional image generation on fMRI voxels and started to use the generative adversarial networks (GANs) to design computational models for this task. Despite the great improvement in previous GAN-based methods, the fidelity and naturalness of the reconstructed images are still unsatisfactory, the reasons include the small number of fMRI data samples and the instability of GAN training. In this study, we propose a new GAN-based Bayesian visual reconstruction model (GAN-BVRM) to avoid the contradiction between naturalness and fidelity in current GAN-based methods. GAN-BVRM is composed of a classifier to decode the categories from fMRI data, a pre-trained conditional generator of the distinguished BigGAN to generate natural images of the specified categories, and a set of encoding models and an evaluator to evaluate the generated images. Composed of neural networks, GAN-BVRM is fully differentiable and can directly generate the reconstructed images by iteratively updating the noise input vector through backpropagation to fit the fMRI voxels. In this process, the decoded categories and encoding models are responsible for the semantic and detailed contents of the reconstructed images, respectively. Experimental results revealed that GAN-BVRM improved the fidelity and naturalness, which validated the advantage of the combining of GANs and Bayesian manner for visual reconstruction. (C) 2020 IBRO. Published by Elsevier Ltd. All rights reserved.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据