4.5 Article

Multimodal processing in face-to-face interactions: A bridging link between psycholinguistics and sensory neuroscience

Related references

Note: Only part of the references are listed.
Article Psychology, Multidisciplinary

Interactionally Embedded Gestalt Principles of Multimodal Human Communication

James P. Trujillo et al.

Summary: Natural human interaction requires us to process and produce various signals, such as speech, gestures, and facial expressions. This article introduces a framework called interactionally embedded, affordance-driven gestalt perception to explain how humans efficiently process multimodal signals. It discusses empirical evidence of gestalt perception principles in unimodal phenomena and proposes a framework where high-level gestalt predictions are updated by sensory input. The article also provides testable predictions from this framework to understand complex, multimodal behaviors in natural social interaction.

PERSPECTIVES ON PSYCHOLOGICAL SCIENCE (2023)

Article Neurosciences

Instantaneous Neural Processing of Communicative Functions Conveyed by Speech Prosody

Rosario Tomasello et al.

Summary: Speech prosody provides important clues about the speaker's communicative intentions. This study used high-density electroencephalography (EEG) to investigate the neurophysiological basis of intonation and speech act understanding, finding that prosodic features are reflected at the neurophysiological level. The results demonstrate that humans can rapidly detect and understand speaker intentions in linguistic interactions through neurophysiological indexes when pragmatic and lexico-semantic information are fully expressed.

CEREBRAL CORTEX (2022)

Article Neurosciences

Spatial and Temporal Context Jointly Modulate the Sensory Response within the Ventral Visual Stream

Tao He et al.

Summary: Both spatial and temporal context are important for visual perception and behavior. This fMRI study found that both forms of context independently modulate neural responses, with stimuli in expected contexts being suppressed compared to unexpected contexts throughout the ventral visual stream.

JOURNAL OF COGNITIVE NEUROSCIENCE (2022)

Review Behavioral Sciences

Two 'what' pathways for action and object recognition

Moritz F. Wurm et al.

Summary: The ventral visual stream is responsible for object recognition, while a pathway in the lateral occipitotemporal cortex plays a critical role in action recognition. The occipitotemporal cortex contains both lateral and ventral pathways for action and object recognition, respectively, which are functionally and anatomically distinct.

TRENDS IN COGNITIVE SCIENCES (2022)

Review Biology

Visual bodily signals as core devices for coordinating minds in interaction

Judith Holler

Summary: This article argues that visual bodily signals play a crucial role in human communication and mind coordination, going beyond referential and propositional meaning. The human communication system is a multimodal, multilayered, multifunctional system that developed and survived due to its flexibility and adaptability. Visual bodily signals, including gestures, facial expressions, and gaze, contribute significantly to pragmatic processes in modern human communication. This contribution is particularly evident when considering non-iconic manual signals, non-manual signals, and signal combinations.

PHILOSOPHICAL TRANSACTIONS OF THE ROYAL SOCIETY B-BIOLOGICAL SCIENCES (2022)

Article Multidisciplinary Sciences

A hierarchy of linguistic predictions during natural language comprehension

Micha Heilbron et al.

Summary: Understanding spoken language requires transforming ambiguous acoustic streams into a hierarchy of representations. The brain uses prediction to guide the interpretation of incoming input, and predictions are ubiquitous in language processing. Predictions at different levels interact with each other.

PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA (2022)

Article Neurosciences

Updating Contextual Sensory Expectations for Adaptive Behavior

Ambra Ferrari et al.

Summary: The brain constructs predictive models of the environment by internalizing statistical regularities in sensory inputs, shaping our perception and reactions. Contextual expectations may need revision as the context changes, but this issue has been neglected in previous research. This fMRI study explores how contextual visual expectations spread throughout the cortical hierarchy and found that the emergence of contextual expectation suppression occurs progressively across cortical areas, with insula, inferior frontal gyrus, and posterior parietal cortex showing the first suppression. These effects are selective to task-relevant expectations.

JOURNAL OF NEUROSCIENCE (2022)

Article Neurosciences

The role of motion in the neural representation of social interactions in the posterior temporal cortex

Julia Landsiedel et al.

Summary: This study confirms that bilateral SI-pSTS plays a central role in dynamic social interaction perception, but is less involved when 'interactiveness' is conveyed solely with static cues. Regions in the social brain, including SI-pSTS and EBA, are sensitive to both motion and interactive content. SI-pSTS is somewhat more tuned to video interactions than is EBA, but both regions show a greater response to social interactions compared to non-interactions and both respond more strongly to videos than static images.

NEUROIMAGE (2022)

Review Multidisciplinary Sciences

The emergent properties of the connected brain

Michel Thiebaut de Schotten et al.

Summary: Brain connections go beyond signal transmission and play a crucial role in behavior and cognition. They determine the brain's functional organization and contribute to understanding brain evolution and cognitive differences. Pathologies affect connections, leading to cognitive dysfunction, and the prediction of long-term symptoms is now based on brain disconnections.

SCIENCE (2022)

Editorial Material Neurosciences

Hyperscanning: Beyond the Hype

Antonia F. de C. Hamilton

Summary: This paper demonstrates how integrating behavioral data and mutual prediction models into hyperscanning studies can advance embodied social neuroscience.

NEURON (2021)

Editorial Material Behavioral Sciences

Hyperscanning Alone Cannot Prove Causality. Multibrain Stimulation Can

Giacomo Novembre et al.

Summary: Brains that work together synchronize and may facilitate social interaction. Studying this phenomenon requires multibrain stimulation rather than just recording brain activity. Promising findings and future horizons in this field are highlighted.

TRENDS IN COGNITIVE SCIENCES (2021)

Review Behavioral Sciences

Evidence for a Third Visual Pathway Specialized for Social Perception

David Pitcher et al.

Summary: According to existing models, primate visual cortex is divided into two functionally distinct pathways, but recent studies have revealed the existence of a third pathway specialized for the dynamic aspects of social perception.

TRENDS IN COGNITIVE SCIENCES (2021)

Article Biochemistry & Molecular Biology

Attention controls multisensory perception via 2 distinct mechanisms at different levels of the cortical hierarchy

Ambra Ferrari et al.

Summary: The study demonstrates that the brain molds multisensory inference through two distinct mechanisms: enhancing visual inputs' reliability and influence in visual and posterior parietal cortices with prestimulus attention to vision, and flexibly combining sensory estimates into spatial representations consistent with Bayesian causal inference in parietal cortices based on poststimulus report. The results show that different neural mechanisms control how signals are combined for perceptual inference at different levels of the cortical hierarchy.

PLOS BIOLOGY (2021)

Review Neurosciences

Perceptual Inference, Learning, and Attention in a Multisensory World

Uta Noppeney

Summary: The brain faces fundamental computational challenges in adaptive behavior within a complex, dynamic, and multisensory world, including inference, decision-making, learning, binding, and attention. Through integrating sensory signals and weighting them based on uncertainties, the brain supports perceptual inference and decision-making, as well as solves the binding problem. Attention mechanisms play a crucial role in computing approximate solutions to the binding problem in naturalistic environments with complex time-varying signals from myriad causes, and the brain dynamically adapts multisensory processing across multiple timescales.

ANNUAL REVIEW OF NEUROSCIENCE, VOL 44, 2021 (2021)

Article Multidisciplinary Sciences

Predicting speech from a cortical hierarchy of event-based time scales

Lea-Maria Schmitt et al.

Summary: The brain incorporates the temporal unfolding of context in a hierarchical manner by sparsely updating contextual representations at event boundaries. Training artificial neural networks and using functional magnetic resonance imaging reveal an event-based surprisal hierarchy evolving along a temporoparietal pathway. Surprisal influences connectivity between neighboring time scales and temporoparietal activity, demonstrating an efficient and contextually diverse network architecture for predictions.

SCIENCE ADVANCES (2021)

Article Multidisciplinary Sciences

Differential spatial computations in ventral and lateral face-selective regions are scaffolded by structural connections

Dawn Finzi et al.

Summary: The study shows that spatial computations in face-selective regions vary across ventral and lateral streams, constrained by connections from early visual areas.

NATURE COMMUNICATIONS (2021)

Article Neurosciences

Facial Signals and Social Actions in Multimodal Face-to-Face Interaction

Naomi Nota et al.

Summary: This study investigates the importance of facial signals in conversations, showing differences in the distribution and timing of facial signals between questioning and responding. Facial signals may critically contribute to the communication of social actions by providing social action-specific visual information.

BRAIN SCIENCES (2021)

Article Neurosciences

Real-Time Eye-to-Eye Contact Is Associated With Cross-Brain Neural Coupling in Angular Gyrus

J. Adam Noah et al.

FRONTIERS IN HUMAN NEUROSCIENCE (2020)

Review Multidisciplinary Sciences

Evaluating the neurophysiological evidence for predictive processing as a model of perception

Kevin S. Walsh et al.

ANNALS OF THE NEW YORK ACADEMY OF SCIENCES (2020)

Review Neurosciences

Using second-person neuroscience to elucidate the mechanisms of social interaction

Elizabeth Redcay et al.

NATURE REVIEWS NEUROSCIENCE (2019)

Article Biochemistry & Molecular Biology

To integrate or not to integrate: Temporal dynamics of hierarchical Bayesian causal inference

Mate Aller et al.

PLOS BIOLOGY (2019)

Article Neurosciences

Dyadic interaction processing in the posterior temporal cortex

Jon Walbrin et al.

NEUROIMAGE (2019)

Review Behavioral Sciences

Multimodal Language Processing in Human Communication

Judith Holler et al.

TRENDS IN COGNITIVE SCIENCES (2019)

Review Multidisciplinary Sciences

The neurobiology of language beyond single-word processing

Peter Hagoort

SCIENCE (2019)

Review Psychology, Mathematical

Virtual reality: A game-changing method for the language sciences

David Peeters

PSYCHONOMIC BULLETIN & REVIEW (2019)

Article Neurosciences

Statistical Learning Signals in Macaque Inferior Temporal Cortex

Peter Kaposvari et al.

CEREBRAL CORTEX (2018)

Review Behavioral Sciences

How Do Expectations Shape Perception?

Floris P. de Lange et al.

TRENDS IN COGNITIVE SCIENCES (2018)

Article Neurosciences

A cross-brain neural mechanism for human-to-human verbal communication

Joy Hirsch et al.

SOCIAL COGNITIVE AND AFFECTIVE NEUROSCIENCE (2018)

Article Psychology, Mathematical

Processing language in face-to-face conversation: Questions with gestures get faster responses

Judith Holler et al.

PSYCHONOMIC BULLETIN & REVIEW (2018)

Article Neurosciences

Auditory, Visual and Audiovisual Speech Processing Streams in Superior Temporal Sulcus

Jonathan H. Venezia et al.

FRONTIERS IN HUMAN NEUROSCIENCE (2017)

Review Neurosciences

Neuroscience-Inspired Artificial Intelligence

Demis Hassabis et al.

NEURON (2017)

Article Neurosciences

Prediction During Natural Language Comprehension

Roel M. Willems et al.

CEREBRAL CORTEX (2016)

Article Behavioral Sciences

The contribution of surprise to the prediction based modulation of fMRI responses

Catarina Amado et al.

NEUROPSYCHOLOGIA (2016)

Article Multidisciplinary Sciences

Perceptual learning of degraded speech by minimizing prediction error

Ediz Sohoglu et al.

PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA (2016)

Article Multidisciplinary Sciences

Brain responses in humans reveal ideal observer-like sensitivity to complex acoustic patterns

Nicolas Barascud et al.

PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA (2016)

Review Neurosciences

Defining Auditory-Visual Objects: Behavioral Tests and Physiological Mechanisms

Jennifer K. Bizley et al.

TRENDS IN NEUROSCIENCES (2016)

Article Biochemistry & Molecular Biology

Prediction Errors but Not Sharpened Signals Simulate Multivoxel fMRI Patterns during Speech Perception

Helen Blank et al.

PLOS BIOLOGY (2016)

Review Behavioral Sciences

An integrative neural model of social perception, action observation, and theory of mind

Daniel Y. -J. Yang et al.

NEUROSCIENCE AND BIOBEHAVIORAL REVIEWS (2015)

Review Behavioral Sciences

The lateral occipitotemporal cortex in action

Angelika Lingnau et al.

TRENDS IN COGNITIVE SCIENCES (2015)

Review Behavioral Sciences

Deconstructing and reconstructing theory of mind

Sara M. Schaafsma et al.

TRENDS IN COGNITIVE SCIENCES (2015)

Article Biochemistry & Molecular Biology

Cortical Hierarchies Perform Bayesian Causal Inference in Multisensory Perception

Tim Rohe et al.

PLOS BIOLOGY (2015)

Article Neurosciences

Rapid neural discrimination of communicative gestures

Elizabeth Redcay et al.

SOCIAL COGNITIVE AND AFFECTIVE NEUROSCIENCE (2015)

Article Psychology, Developmental

The Relationship Between Infant Holdout and Gives, and Pointing

Thea Cameron-Faulkner et al.

INFANCY (2015)

Letter Biochemistry & Molecular Biology

Temporal prediction errors in visual and auditory cortices

Hweeling Lee et al.

CURRENT BIOLOGY (2014)

Article Biochemistry & Molecular Biology

Combined TMS and fMRI Reveal Dissociable Cortical Pathways for Dynamic and Static Face Perception

David Pitcher et al.

CURRENT BIOLOGY (2014)

Article Biology

The origin of human multi-modal communication

Stephen C. Levinson et al.

PHILOSOPHICAL TRANSACTIONS OF THE ROYAL SOCIETY B-BIOLOGICAL SCIENCES (2014)

Review Biology

Hearing and seeing meaning in speech and gesture: insights from brain and behaviour

Asli Ozyurek

PHILOSOPHICAL TRANSACTIONS OF THE ROYAL SOCIETY B-BIOLOGICAL SCIENCES (2014)

Article Neurosciences

Perception of affective and linguistic prosody: an ALE meta-analysis of neuroimaging studies

Michel Belyk et al.

SOCIAL COGNITIVE AND AFFECTIVE NEUROSCIENCE (2014)

Review Psychology

Intersensory binding across space and time: A tutorial review

Lihan Chen et al.

ATTENTION PERCEPTION & PSYCHOPHYSICS (2013)

Article Psychology, Biological

Whatever next? Predictive brains, situated agents, and the future of cognitive science

Andy Clark

BEHAVIORAL AND BRAIN SCIENCES (2013)

Article Neurosciences

Theory of Mind: A Neural Prediction Problem

Jorie Koster-Hale et al.

NEURON (2013)

Article Neurosciences

The thing that should not be: predictive coding and the uncanny valley in perceiving human and humanoid robot actions

Ayse Pinar Saygin et al.

SOCIAL COGNITIVE AND AFFECTIVE NEUROSCIENCE (2012)

Review Behavioral Sciences

The cortical language circuit: from auditory perception to sentence comprehension

Angela D. Friederici

TRENDS IN COGNITIVE SCIENCES (2012)

Review Psychology

The Neural Bases of Social Cognition and Story Comprehension

Raymond A. Mar

ANNUAL REVIEW OF PSYCHOLOGY, VOL 62 (2011)

Article Neurosciences

The Contributions of Transient and Sustained Response Codes to Audiovisual Integration

Sebastian Werner et al.

CEREBRAL CORTEX (2011)

Article Biochemistry & Molecular Biology

Voice Cells in the Primate Temporal Lobe

Catherine Perrodin et al.

CURRENT BIOLOGY (2011)

Article Neurosciences

The auditory cortex mediates the perceptual effects of acoustic temporal expectation

Santiago Jaramillo et al.

NATURE NEUROSCIENCE (2011)

Article Multidisciplinary Sciences

Statistical learning of visual transitions in monkey inferotemporal cortex

Travis Meyer et al.

PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA (2011)

Article Biochemistry & Molecular Biology

Visual Enhancement of the Information Representation in Auditory Cortex

Christoph Kayser et al.

CURRENT BIOLOGY (2010)

Article Neurosciences

Stimulus Predictability Reduces Responses in Primary Visual Cortex

Arjen Alink et al.

JOURNAL OF NEUROSCIENCE (2010)

Review Neurosciences

The free-energy principle: a unified brain theory?

Karl J. Friston

NATURE REVIEWS NEUROSCIENCE (2010)

Review Behavioral Sciences

Causal inference in perception

Ladan Shams et al.

TRENDS IN COGNITIVE SCIENCES (2010)

Article Neurosciences

The Differing Impact of Multisensory and Unisensory Integration on Behavior

Guy Gingras et al.

JOURNAL OF NEUROSCIENCE (2009)

Review Clinical Neurology

Hallucinations in Parkinson disease

Nico J. Diederich et al.

NATURE REVIEWS NEUROLOGY (2009)

Article Neurosciences

Visual modulation of neurons in auditory cortex

Christoph Kayser et al.

CEREBRAL CORTEX (2008)

Review Neurosciences

Superior Temporal Sulcus-It's My Area: Or Is It?

Grit Hein et al.

JOURNAL OF COGNITIVE NEUROSCIENCE (2008)

Article Neurosciences

Neural repetition suppression reflects fulfilled perceptual expectations

Christopher Summerfield et al.

NATURE NEUROSCIENCE (2008)

Article Neurosciences

A voice region in the monkey brain

Christopher I. Petkov et al.

NATURE NEUROSCIENCE (2008)

Article Neurosciences

Neural correlates of the processing of co-speech gestures

Henning Holle et al.

NEUROIMAGE (2008)

Article Neurosciences

Touch, sound and vision in human superior temporal sulcus

Michael S. Beauchamp et al.

NEUROIMAGE (2008)

Article Multidisciplinary Sciences

Causal Inference in Multisensory Perception

Konrad P. Koerding et al.

PLOS ONE (2007)

Article Neurosciences

Multisensory interactions within human primary cortices revealed by BOLD dynamics

Roberto Martuzzi et al.

CEREBRAL CORTEX (2007)

Review Neurosciences

Opinion - The cortical organization of speech processing

Gregory Hickok et al.

NATURE REVIEWS NEUROSCIENCE (2007)

Article Ophthalmology

Sources of somatosensory input to the caudal belt areas of auditory cortex

Troy A. Hackett et al.

PERCEPTION (2007)

Review Anatomy & Morphology

Do early sensory cortices integrate cross-modal information?

Christoph Kayser et al.

BRAIN STRUCTURE & FUNCTION (2007)

Review Neurosciences

The neural basis of mentalizing

Chris D. Frith et al.

NEURON (2006)

Article Neurosciences

Touch activates human auditory cortex

Martin Schurmann et al.

NEUROIMAGE (2006)

Review Neurosciences

Cortical connections of the auditory cortex in marmoset monkeys: Core and medial belt regions

LA de la Mothe et al.

JOURNAL OF COMPARATIVE NEUROLOGY (2006)

Review Behavioral Sciences

Is neocortex essentially multisensory?

Asif A. Ghazanfar et al.

TRENDS IN COGNITIVE SCIENCES (2006)

Article Neurosciences

Integration of touch and sound in auditory cortex

C Kayser et al.

NEURON (2005)

Article Biology

A theory of cortical responses

KJ Friston

PHILOSOPHICAL TRANSACTIONS OF THE ROYAL SOCIETY B-BIOLOGICAL SCIENCES (2005)

Review Neurosciences

See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex

MS Beauchamp

CURRENT OPINION IN NEUROBIOLOGY (2005)

Review Behavioral Sciences

On Broca, brain, and binding: a new framework

P Hagoort

TRENDS IN COGNITIVE SCIENCES (2005)

Article Neurosciences

Integration of letters and speech sounds in the human brain

N van Atteveldt et al.

NEURON (2004)

Article Neurosciences

Multisensory integration: methodological approaches and emerging principles in the human brain

GA Calvert et al.

JOURNAL OF PHYSIOLOGY-PARIS (2004)

Article Neurosciences

Somatosensory input to auditory association cortex in the macaque monkey

CE Schroeder et al.

JOURNAL OF NEUROPHYSIOLOGY (2001)

Article Multidisciplinary Sciences

Mechanisms and streams for processing of what and where in auditory cortex

JP Rauschecker et al.

PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA (2000)

Article Computer Science, Artificial Intelligence

Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping

JJ Foxe et al.

COGNITIVE BRAIN RESEARCH (2000)

Review Behavioral Sciences

Social perception from visual cues: role of the STS region

T Allison et al.

TRENDS IN COGNITIVE SCIENCES (2000)

Review Behavioral Sciences

Towards a functional neuroanatomy of speech perception

G Hickok et al.

TRENDS IN COGNITIVE SCIENCES (2000)