期刊
LANGUAGE COGNITION AND NEUROSCIENCE
卷 -, 期 -, 页码 -出版社
ROUTLEDGE JOURNALS, TAYLOR & FRANCIS LTD
DOI: 10.1080/23273798.2023.2295499
关键词
Multimodality; co-speech gestures; surprisal; word frequency; regression ERP; lexical retrieval; semantic prediction; N400
This study investigates the impact of gestures on lexical retrieval and semantic prediction during the processing of naturalistic multimodal stimuli. The results suggest that co-speech gestures have a modulatory effect on word frequency and surprisal, facilitating lexical retrieval and potentially semantic prediction.
During daily communication, visual cues such as gestures accompany the speech signal and facilitate semantic processing. However, how gestures impact lexical retrieval and semantic prediction, especially in a naturalistic setting, remains unclear. Here, participants watched a naturalistic multimodal narrative, where an actor narrated a story and spontaneously produced co-speech gestures. For all content words, word frequency and lexical surprisal were regressed against the EEG using temporal response functions (TFRs), which were fitted separately, additively, and interactively for words accompanied and not accompanied by gestures. Results from our analyses suggest a robust modulation effect of gesture on the frequency-dependent regression N400. Besides, we also observed some evidence of modulative effect of gesture on the surprisal-N400 effect based on the single-predictor model. Our finding thus suggests that, on a neural level, the presence of co-speech gestures facilitates lexical retrieval and potentially semantic prediction during the processing of naturalistic multimodal stimuli.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据