Journal
NEUROSCIENCE BULLETIN
Volume 37, Issue 3, Pages 369-379Publisher
SPRINGER
DOI: 10.1007/s12264-020-00613-4
Keywords
Brain decoding; fMRI; Deep learning
Categories
Funding
- National Natural Science Foundation of China [61773094, 61533006, U1808204, 31730039, 31671133, 61876114]
- Ministry of Science and Technology of China [2015CB351701]
- National Major Scientific Instruments and Equipment Development Project [ZDYZ2015-2]
- Chinese Academy of Sciences Strategic Priority Research Program B grant [XDB32010300]
Ask authors/readers for more resources
The proposed deep learning-based framework consists of a latent feature extractor, a latent feature decoder, and a natural image generator, aiming to accurately reconstruct natural images from brain activity by extracting and predicting latent features of natural images. This approach shows promise for decoding brain activity, achieving comparable reproduction of presented images in both high-level semantic category information and low-level pixel information.
Brain decoding based on functional magnetic resonance imaging has recently enabled the identification of visual perception and mental states. However, due to the limitations of sample size and the lack of an effective reconstruction model, accurate reconstruction of natural images is still a major challenge. The current, rapid development of deep learning models provides the possibility of overcoming these obstacles. Here, we propose a deep learning-based framework that includes a latent feature extractor, a latent feature decoder, and a natural image generator, to achieve the accurate reconstruction of natural images from brain activity. The latent feature extractor is used to extract the latent features of natural images. The latent feature decoder predicts the latent features of natural images based on the response signals from the higher visual cortex. The natural image generator is applied to generate reconstructed images from the predicted latent features of natural images and the response signals from the visual cortex. Quantitative and qualitative evaluations were conducted with test images. The results showed that the reconstructed image achieved comparable, accurate reproduction of the presented image in both high-level semantic category information and low-level pixel information. The framework we propose shows promise for decoding the brain activity.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available