期刊
ROYAL SOCIETY OPEN SCIENCE
卷 9, 期 4, 页码 -出版社
ROYAL SOC
DOI: 10.1098/rsos.211489
关键词
kinematics; gesture; speech; compensation; multimodal
This study finds that in human communication, speech can compensate when the visual channel is disrupted, contradicting the notion that gesture plays a secondary role. Both gesture and speech dynamically adapt to meet communicative needs, highlighting the flexible nature of multi-modal communicative behaviors.
In human communication, when the speech is disrupted, the visual channel (e.g. manual gestures) can compensate to ensure successful communication. Whether speech also compensates when the visual channel is disrupted is an open question, and one that significantly bears on the status of the gestural modality. We test whether gesture and speech are dynamically co-adapted to meet communicative needs. To this end, we parametrically reduce visibility during casual conversational interaction and measure the effects on speakers' communicative behaviour using motion tracking and manual annotation for kinematic and acoustic analyses. We found that visual signalling effort was flexibly adapted in response to a decrease in visual quality (especially motion energy, gesture rate, size, velocity and hold-time). Interestingly, speech was also affected: speech intensity increased in response to reduced visual quality (particularly in speech-gesture utterances, but independently of kinematics). Our findings highlight that multi-modal communicative behaviours are flexibly adapted at multiple scales of measurement and question the notion that gesture plays an inferior role to speech.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据