An experimental study of speech/gesture interactions and distance encoding

This paper explores the possible encoding of distance information in vocal and manual pointing and its relationship with the linguistic structure of deictic words, as well as speech/gesture cooperation within the process of deixis. Two experiments required participants to point at and/or name a close or distant target, with speech only, with gesture only, or with speech gesture. Acoustic, articulatory, and manual data were recorded. We investigated the interaction between vocal and manual pointing, with respect to the distance to the target. There are two major findings. First, distance significantly affects both articulatory and manual pointing, since participants perform larger vocal and manual gestures to designate a more distant target. Second, modality influences both deictic speech and gesture, since pointing is more emphatic in unimodal use of either over bimodal use of both, to compensate for the loss of the other mode. These findings suggest that distance is encoded in both vocal and manual pointing. We also demonstrate that the correlates of distance encoding in the vocal modality can be related to the typology of deictic words. Finally, our data suggest a two-way interaction between speech and gesture, and support the hypothesis that these two modalities are cooperating within a single communication system. (C) 2012 Elsevier B.V. All rights reserved.