The Perceptual and Cognitive Role of Visual and Auditory Channels in Conveying Emotional Information

In expressing emotional states, we generate a synaesthesic experience in our interlocutor when we transmit information about our feelings by the simultaneous use of several sensorial channels. These are referred to as verbal (the semantic content of the message) and non-verbal (gesture, gaze, tonal expression) modalities. From an engineering point of view the transmission of the same information by more than one sensorial channel should be considered as redundant. Is this true or does each channel transmit a specific piece of information? How much emotional information is transmitted by each channel and which plays a major role? As an attempt to address these questions, here we present a comparative analysis of the subjective perception of emotional states by the visual and auditory channels considered either singularly or in combination, always in the non-verbal modality. The results reveal that the audio and visual components of emotional messages convey much the same amount of information either separately or in combination, hence suggesting that each channel performs a robust encoding of the emotional features. Redundancy probably facilitates the recovery of emotional information in case one of the channels is impaired. This conclusion is challenged by language cultural specificity, since when tested on a foreign language the addressers rely more on visual information.