4.7 Article

Multimodal deep neural decoding reveals highly resolved spatiotemporal profile of visual object representation in humans

期刊

NEUROIMAGE
卷 275, 期 -, 页码 -

出版社

ACADEMIC PRESS INC ELSEVIER SCIENCE
DOI: 10.1016/j.neuroimage.2023.120164

关键词

fMRI; EEG; Deep neural network; Visual system

向作者/读者索取更多资源

Perception and categorization of objects in visual scenes can provide a better understanding of the surrounding situation. This study explored the spatial and temporal organization of visual object representations by using concurrent fMRI and EEG data, combined with deep neural networks. The results demonstrate the ability of multimodal deep learning to efficiently classify visual objects and sub-categories, and reveal the mechanisms of object categorization in brain-wide regions.
Perception and categorization of objects in a visual scene are essential to grasp the surrounding situation. Re-cently, neural decoding schemes, such as machine learning in functional magnetic resonance imaging (fMRI), has been employed to elucidate the underlying neural mechanisms. However, it remains unclear as to how spatially distributed brain regions temporally represent visual object categories and sub-categories. One promising strategy to address this issue is neural decoding with concurrently obtained neural response data of high spatial and tem-poral resolution. In this study, we explored the spatial and temporal organization of visual object representations using concurrent fMRI and electroencephalography (EEG), combined with neural decoding using deep neural networks (DNNs). We hypothesized that neural decoding by multimodal neural data with DNN would show high classification performance in visual object categorization (faces or non-face objects) and sub-categorization within faces and objects. Visualization of the fMRI DNN was more sensitive than that in the univariate approach and revealed that visual categorization occurred in brain-wide regions. Interestingly, the EEG DNN valued the earlier phase of neural responses for categorization and the later phase of neural responses for sub-categorization. Com-bination of the two DNNs improved the classification performance for both categorization and sub-categorization compared with fMRI DNN or EEG DNN alone. These deep learning-based results demonstrate a categorization principle in which visual objects are represented in a spatially organized and coarse-to-fine manner, and provide strong evidence of the ability of multimodal deep learning to uncover spatiotemporal neural machinery in sensory processing.

作者

我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。

评论

主要评分

4.7
评分不足

次要评分

新颖性
-
重要性
-
科学严谨性
-
评价这篇论文

推荐

暂无数据
暂无数据