期刊
PROCEEDINGS OF THE ROYAL SOCIETY B-BIOLOGICAL SCIENCES
卷 288, 期 1955, 页码 -出版社
ROYAL SOC
DOI: 10.1098/rspb.2021.0500
关键词
comprehensions; prosody; gesture; mouth; surprisal; N400
资金
- European Research Council Advanced Grant [743035]
- Royal Society Wolfson Research Merit Award [WRM\R3\170016]
- European Research Council (ERC) [743035] Funding Source: European Research Council (ERC)
Two studies indicate that multimodal cues are crucial for comprehension, with brain responses to words being influenced by the informativeness of co-occurring cues.
The ecology of human language is face-to-face interaction, comprising cues such as prosody, co-speech gestures and mouth movements. Yet, the multimodal context is usually stripped away in experiments as dominant paradigms focus on linguistic processing only. In two studies we presented video-clips of an actress producing naturalistic passages to participants while recording their electroencephalogram. We quantified multimodal cues (prosody, gestures, mouth movements) and measured their effect on a well-established electroencephalographic marker of processing load in comprehension (N400). We found that brain responses to words were affected by informativeness of co-occurring multimodal cues, indicating that comprehension relies on linguistic and non-linguistic cues. Moreover, they were affected by interactions between the multimodal cues, indicating that the impact of each cue dynamically changes based on the informativeness of other cues. Thus, results show that multimodal cues are integral to comprehension, hence, our theories must move beyond the limited focus on speech and linguistic processing.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据