The Effect of Audio-Visual Smiles on Social Influence in a Cooperative Human-Agent Interaction Task

Emotional expressivity is essential for human interactions, informing both perception and decision-making. Here, we examine whether creating an audio-visual emotional channel mismatch influences decision-making in a cooperative task with a virtual character. We created a virtual character that was either congruent in its emotional expression (smiling in the face and voice) or incongruent (smiling in only one channel). People (N = 98) evaluated the character in terms of valence and arousal in an online study; then, visitors in a museum played the lunar survival task with the character over three experiments (N = 597, 78, 101, respectively). Exploratory results suggest that multi-modal expressions are perceived, and reacted upon, differently than unimodal expressions, supporting previous theories of audio-visual integration.

Automated summary

The study found that creating an audio-visual emotional channel mismatch may influence decision-making in a cooperative task; multi-modal expressions are perceived and reacted upon differently than unimodal expressions.