Multimodal Integration

Theoretical and practical motives alike have prompted recent investigations of multimodal speech perception. Theoretically, multimodal studies have extended the conceptualization of perceptual organization beyond the familiar modality-bound accounts deriving from Gestalt psychology. Practically, such investigations have been driven by a need to understand the proficiency of multimodal speech perception using an electrocochlear prosthesis for hearing. In each domain, studies have shown that perceptual organization of speech can occur even when the perceiver's auditory experience departs from natural speech qualities. Accordingly, our research examined auditor-visual multimodal integration of videotaped faces and selected acoustic constituents of speech signals, each realized as a single sinewave tone accompanying a video image of an articulating face. The single tone reproduced the frequency and amplitude of the phonatory cycle or of one of the lower three oral formants. Our res...

When multilingual communication through a speech-to-speech translation system is supported by multimodal features, e.g. pen-based gestures, the following issues arise concerning the nature of the supported communication: a) to what extend does multilingual communication differ from ‘ordinary’ monolingual communication with respect to the dialogue structure and the communicative strategies used by participants; b) the patterns of integration between speech and gestures. Building on the outcomes of a previous work, we present results from a study aimed at addressing those issues. The initial findings confirm that multilingual communication, and the way in which it is realized by actual systems (e.g., with or without the push-to-talk mode) affects the form and structure of the conversation.

As described by Oviatt et al.(1997), when two or more modalities work together, the integration techniques used for combining different modalities into a whole system is very important. The integration techniques are the main determinants in guiding the design of the multimodal system. To resolve a map-based multimodal system, we propose to use natural language processing (NLP) to identify and describe the syntax and the semantics of prepositions within the spoken speech, in particular, when it relates to maps or directions. ...

In this letter we describe a hippocampo-cortical model of spatial processing and navigation based on a cascade of increasingly complex associative processes that are also relevant for other hippocampal functions such as episodic memory. Associative learning of different types and the related pattern encoding-recognition take place at three successive levels: (1) an object location level, which computes the landmarks from merged multimodal sensory inputs in the parahippocampal cortices; (2) a subject location level, which computes place fields by combination of local views and movement-related information in the entorhinal cortex; and (3) a spatiotemporal level, which computes place transitions from contiguous place fields in the CA3-CA1 region, which form building blocks for learning temporospatial sequences. At the cell population level, superficial entorhinal place cells encode spatial, context-independent maps as landscapes of activity; populations of transition cells in the CA3-...

Nowadays, several types of brain imaging device are available to provide images of the functional activity of the cerebral cortex based on hemodynamic, metabolic, or electromagnetic measurements. However, static images of brain regions activated during particular tasks do not convey the information of how these regions communicate with each other. In this study, advanced methods for the estimation of cortical connectivity from combined high-resolution electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) data are presented. These methods include a subject's multicompartment head model (scalp, skull, dura mater, cortex) constructed from individual magnetic resonance images, multidipole source model, and regularized linear inverse source estimates of cortical current density. Determination of the priors in the resolution of the linear inverse problem was performed with the use of information from the hemodynamic responses of the cortical areas as revealed by block-designed (strength of activated voxels) fMRI. We estimate functional cortical connectivity by computing the directed transfer function (DTF) on the estimated cortical current density waveforms in regions of interest (ROIs) on the modeled cortical mantle. The proposed method was able to unveil the direction of the information flow between the cortical regions of interest, as it is directional in nature. Furthermore, this method allows to detect changes in the time course of information flow between cortical regions in different frequency bands. The reliability of these techniques was further demonstrated by elaboration of high-resolution EEG and fMRI signals collected during visually triggered finger movements in four healthy subjects. Connectivity patterns estimated for this task reveal an involvement of right parietal and bilateral premotor and prefrontal cortical areas. This cortical region involvement resembles that revealed in previous studies where visually triggered finger movements were analyzed with the use of separate EEG or fMRI measurements.

In this paper, advanced methods for the modeling of human cortical activity from combined high-resolution electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) data are reviewed. These methods include a subject's multicompartment head model (scalp, skull, dura mater, cortex) constructed from magnetic resonance images, multidipole source model, and regularized linear inverse source estimates. Determination of the priors in the resolution of the linear inverse problem was performed with the use of information from the hemodynamic responses of the cortical areas as revealed by block-designed fMRI.

We investigated the interactions between visual, tactile and auditory sensory signals for the perception of sequences of events. Sequences of flashes, taps and beeps were presented simultaneously. For each session, subjects were instructed to count the number of events presented in one modality (Target) and to ignore the stimuli presented in the other modalities (Background). The number of events presented