Journal
PROCEEDINGS OF THE ROYAL SOCIETY B-BIOLOGICAL SCIENCES
Volume 288, Issue 1955, Pages -Publisher
ROYAL SOC
DOI: 10.1098/rspb.2021.0500
Keywords
comprehensions; prosody; gesture; mouth; surprisal; N400
Categories
Funding
- European Research Council Advanced Grant [743035]
- Royal Society Wolfson Research Merit Award [WRM\R3\170016]
- European Research Council (ERC) [743035] Funding Source: European Research Council (ERC)
Ask authors/readers for more resources
Two studies indicate that multimodal cues are crucial for comprehension, with brain responses to words being influenced by the informativeness of co-occurring cues.
The ecology of human language is face-to-face interaction, comprising cues such as prosody, co-speech gestures and mouth movements. Yet, the multimodal context is usually stripped away in experiments as dominant paradigms focus on linguistic processing only. In two studies we presented video-clips of an actress producing naturalistic passages to participants while recording their electroencephalogram. We quantified multimodal cues (prosody, gestures, mouth movements) and measured their effect on a well-established electroencephalographic marker of processing load in comprehension (N400). We found that brain responses to words were affected by informativeness of co-occurring multimodal cues, indicating that comprehension relies on linguistic and non-linguistic cues. Moreover, they were affected by interactions between the multimodal cues, indicating that the impact of each cue dynamically changes based on the informativeness of other cues. Thus, results show that multimodal cues are integral to comprehension, hence, our theories must move beyond the limited focus on speech and linguistic processing.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available