A long tradition of research including classical rhetoric, esthetics and poetics theory, formalis... more A long tradition of research including classical rhetoric, esthetics and poetics theory, formalism and structuralism, as well as current perspectives in (neuro)cognitive poetics has investigated structural and functional aspects of literature reception. Despite a wealth of literature published in specialized journals like Poetics, however, still little is known about how the brain processes and creates literary and poetic texts. Still, such stimulus material might be suited better than other genres for demonstrating the complexities with which our brain constructs the world in and around us, because it unifies thought and language, music and imagery in a clear, manageable way, most often with play, pleasure, and emotion (Schrott and Jacobs, 2011). In this paper, I discuss methods and models for investigating the neuronal and cognitive-affective bases of literary reading together with pertinent results from studies on poetics, text processing, emotion, or neuroaesthetics, and outline current challenges and future perspectives. Aesthetic value, then, is like the wind—we know of its existence only through its effects (Iser, 1976).
While watching movies, the brain integrates the visual information and the musical soundtrack int... more While watching movies, the brain integrates the visual information and the musical soundtrack into a coherent percept. Multisensory integration can lead to emotion elicitation on which soundtrack valences may have a modulatory impact. Here, dynamic kissing scenes from romantic comedies were presented to 22 participants (13 females) during functional magnetic resonance imaging scanning. The kissing scenes were either accompanied by happy music, sad music or no music. Evidence from cross-modal studies motivated a predefined three-region network for multisensory integration of emotion, consisting of fusiform gyrus (FG), amygdala (AMY) and anterior superior temporal gyrus (aSTG). The interactions in this network were investigated using dynamic causal models of effective connectivity. This revealed bilinear modulations by happy and sad music with suppression effects on the connectivity from FG and AMY to aSTG. Non-linear dynamic causal modeling showed a suppressive gating effect of aSTG on fusiform–amygdalar connectivity. In conclusion, fusiform to amygdala coupling strength is modulated via feedback through aSTG as region for multisensory integration of emotional material. This mechanism was emotion-specific and more pronounced for sad music. Therefore, soundtrack valences may modulate emotion elicitation in movies by differentially changing preprocessed visual information to the amygdala.
The temporal pole (TP) has been associated with diverse functions of social cognition and emotion... more The temporal pole (TP) has been associated with diverse functions of social cognition and emotion processing. Although the underlying mechanism remains elusive, one possibility is that TP acts as domain-general hub integrating socioemotional information. To test this, 26 participants were presented with 60 empathy-evoking film clips during fMRI scanning. The film clips were preceded by a linguistic sad or neutral context and half of the clips were accompanied by sad music. In line with its hypothesized role, TP was involved in the processing of sad context and furthermore tracked participants’ empathic concern. To examine the neuromodulatory impact of TP, we applied nonlinear dynamic causal modeling to a multisensory integration network from previous work consisting of superior temporal gyrus (STG), fusiform gyrus (FG), and amygdala, which was extended by an additional node in the TP. Bayesian model comparison revealed a gating of STG and TP on fusiform–amygdalar coupling and an increase of TP to FG connectivity during the integration of contextual information. Moreover, these backward projections were strengthened by emotional music. The findings indicate that during social cognition, TP integrates information from different modalities and top-down modulates lower-level perceptual areas in the ventral visual stream as a function of integration demands.
Abstract Previous research using event-related brain potentials (ERPs) suggested that phonologica... more Abstract Previous research using event-related brain potentials (ERPs) suggested that phonological processing in visual word recognition occurs rather late, typically after semantic or syntactic processing. Here, we show that phonological activation in visual word recognition can be observed much earlier. Using a lexical decision task, we show that ERPs to pseudohomophones (PsHs)(eg, ROZE) differed from well-matched spelling controls (eg, ROFE) as early as 150 ms (P150) after stimulus onset.
During reading, the probability of refixations increases and the duration of first fixations decr... more During reading, the probability of refixations increases and the duration of first fixations decreases with growing distance of the initial fixation position from a word's center (ie, the optimal viewing position, OVP). The question, whether or not refixation-OVP and firstfixation duration inverted-OVP curves are modulated by the lexical characteristics of the actually fixated stimulus is still a matter of debate.
Exploration of the real world usually expresses itself through a perceptual behaviour that is com... more Exploration of the real world usually expresses itself through a perceptual behaviour that is complex and adaptive—an interplay between external visual and internal cognitive states. However, up to now, the measurement of electrophysiological correlates of cognitive processes has been limited to situations, in which the experimental setting confined visual exploration to the mere reception of a strict serial order of events.
Abstract It is hypothesized that written languages differ in the preferred grain size of units th... more Abstract It is hypothesized that written languages differ in the preferred grain size of units that emerge during reading acquisition. Smaller units (graphemes, phonemes) are thought to play a dominant role in relatively consistent orthographies (eg, German), whereas larger units (bodies, rhymes) are thought to be more important in relatively inconsistent orthographies (eg, English).
Recent neurocognitive studies of visual word recognition provide information about neuronal netwo... more Recent neurocognitive studies of visual word recognition provide information about neuronal networks correlated with processes involved in lexical access and their time course (eg,[Holcomb, Ph. J., Grainger J. and O'Rourke, T.(2002). An Electrophysiological Study of the Effects of Orthographic Neighborhood Size on Printed Word Perception, J. of Cogn. Neurosci. 14 938–950; Binder, JR, McKiernan, KA, Parsons, ME, Westbury, CF, Possing, ET, Kaufman, JN and Buchanan, L.(2003).
Zusammenfassung. Vor zehn Jahren wurde eine Reform der deutschen Orthografie beschlossen. Anhand ... more Zusammenfassung. Vor zehn Jahren wurde eine Reform der deutschen Orthografie beschlossen. Anhand einer Blickbewegungsstudie, an der sowohl erwachsene Leser als auch Kinder teilnahmen, sollte untersucht werden, welche Auswirkungen die Rechtschreibreform auf das Lesen hatte. Die Ergebnisse liefern erste Hinweise dafür, dass die Rechtschreibreform die Worterkennung insgesamt nicht beeinträchtigt hat.
The present study investigated the nature of the inhibitory syllable frequency effect, recently r... more The present study investigated the nature of the inhibitory syllable frequency effect, recently reported for normal readers, in a German-speaking dyslexic patient. The reading impairment was characterized as a severe deficit in naming single letters or words in the presence of spared lexical processing of visual word forms.
Many neurocognitive studies investigated the neural correlates of visual word recognition, some o... more Many neurocognitive studies investigated the neural correlates of visual word recognition, some of which manipulated the orthographic neighborhood density of words and nonwords believed to influence the activation of orthographically similar representations in a hypothetical mental lexicon. Previous neuroimaging research failed to find evidence for such global lexical activity associated with neighborhood density. Rather, effects were interpreted to reflect semantic or domain general processing. The present fMRI study revealed effects of lexicality, orthographic neighborhood density and a lexicality by orthographic neighborhood density interaction in a silent reading task. For the first time we found greater activity for words and nonwords with a high number of neighbors. We propose that this activity in the dorsomedial prefrontal cortex reflects activation of orthographically similar codes in verbal working memory thus providing evidence for global lexical activity as the basis of the neighborhood density effect. The interaction of lexicality by neighborhood density in the ventromedial prefrontal cortex showed lower activity in response to words with a high number compared to nonwords with a high number of neighbors. In the light of these results the facilitatory effect for words and inhibitory effect for nonwords with many neighbors observed in previous studies can be understood as being due to the operation of a fast-guess mechanism for words and a temporal deadline mechanism for nonwords as predicted by models of visual word recognition. Furthermore, we propose that the lexicality effect with higher activity for words compared to nonwords in inferior parietal and middle temporal cortex reflects the operation of an identification mechanism based on local lexico-semantic activity.
A long tradition of research including classical rhetoric, esthetics and poetics theory, formalis... more A long tradition of research including classical rhetoric, esthetics and poetics theory, formalism and structuralism, as well as current perspectives in (neuro)cognitive poetics has investigated structural and functional aspects of literature reception. Despite a wealth of literature published in specialized journals like Poetics, however, still little is known about how the brain processes and creates literary and poetic texts. Still, such stimulus material might be suited better than other genres for demonstrating the complexities with which our brain constructs the world in and around us, because it unifies thought and language, music and imagery in a clear, manageable way, most often with play, pleasure, and emotion (Schrott and Jacobs, 2011). In this paper, I discuss methods and models for investigating the neuronal and cognitive-affective bases of literary reading together with pertinent results from studies on poetics, text processing, emotion, or neuroaesthetics, and outline current challenges and future perspectives. Aesthetic value, then, is like the wind—we know of its existence only through its effects (Iser, 1976).
While watching movies, the brain integrates the visual information and the musical soundtrack int... more While watching movies, the brain integrates the visual information and the musical soundtrack into a coherent percept. Multisensory integration can lead to emotion elicitation on which soundtrack valences may have a modulatory impact. Here, dynamic kissing scenes from romantic comedies were presented to 22 participants (13 females) during functional magnetic resonance imaging scanning. The kissing scenes were either accompanied by happy music, sad music or no music. Evidence from cross-modal studies motivated a predefined three-region network for multisensory integration of emotion, consisting of fusiform gyrus (FG), amygdala (AMY) and anterior superior temporal gyrus (aSTG). The interactions in this network were investigated using dynamic causal models of effective connectivity. This revealed bilinear modulations by happy and sad music with suppression effects on the connectivity from FG and AMY to aSTG. Non-linear dynamic causal modeling showed a suppressive gating effect of aSTG on fusiform–amygdalar connectivity. In conclusion, fusiform to amygdala coupling strength is modulated via feedback through aSTG as region for multisensory integration of emotional material. This mechanism was emotion-specific and more pronounced for sad music. Therefore, soundtrack valences may modulate emotion elicitation in movies by differentially changing preprocessed visual information to the amygdala.
The temporal pole (TP) has been associated with diverse functions of social cognition and emotion... more The temporal pole (TP) has been associated with diverse functions of social cognition and emotion processing. Although the underlying mechanism remains elusive, one possibility is that TP acts as domain-general hub integrating socioemotional information. To test this, 26 participants were presented with 60 empathy-evoking film clips during fMRI scanning. The film clips were preceded by a linguistic sad or neutral context and half of the clips were accompanied by sad music. In line with its hypothesized role, TP was involved in the processing of sad context and furthermore tracked participants’ empathic concern. To examine the neuromodulatory impact of TP, we applied nonlinear dynamic causal modeling to a multisensory integration network from previous work consisting of superior temporal gyrus (STG), fusiform gyrus (FG), and amygdala, which was extended by an additional node in the TP. Bayesian model comparison revealed a gating of STG and TP on fusiform–amygdalar coupling and an increase of TP to FG connectivity during the integration of contextual information. Moreover, these backward projections were strengthened by emotional music. The findings indicate that during social cognition, TP integrates information from different modalities and top-down modulates lower-level perceptual areas in the ventral visual stream as a function of integration demands.
Abstract Previous research using event-related brain potentials (ERPs) suggested that phonologica... more Abstract Previous research using event-related brain potentials (ERPs) suggested that phonological processing in visual word recognition occurs rather late, typically after semantic or syntactic processing. Here, we show that phonological activation in visual word recognition can be observed much earlier. Using a lexical decision task, we show that ERPs to pseudohomophones (PsHs)(eg, ROZE) differed from well-matched spelling controls (eg, ROFE) as early as 150 ms (P150) after stimulus onset.
During reading, the probability of refixations increases and the duration of first fixations decr... more During reading, the probability of refixations increases and the duration of first fixations decreases with growing distance of the initial fixation position from a word's center (ie, the optimal viewing position, OVP). The question, whether or not refixation-OVP and firstfixation duration inverted-OVP curves are modulated by the lexical characteristics of the actually fixated stimulus is still a matter of debate.
Exploration of the real world usually expresses itself through a perceptual behaviour that is com... more Exploration of the real world usually expresses itself through a perceptual behaviour that is complex and adaptive—an interplay between external visual and internal cognitive states. However, up to now, the measurement of electrophysiological correlates of cognitive processes has been limited to situations, in which the experimental setting confined visual exploration to the mere reception of a strict serial order of events.
Abstract It is hypothesized that written languages differ in the preferred grain size of units th... more Abstract It is hypothesized that written languages differ in the preferred grain size of units that emerge during reading acquisition. Smaller units (graphemes, phonemes) are thought to play a dominant role in relatively consistent orthographies (eg, German), whereas larger units (bodies, rhymes) are thought to be more important in relatively inconsistent orthographies (eg, English).
Recent neurocognitive studies of visual word recognition provide information about neuronal netwo... more Recent neurocognitive studies of visual word recognition provide information about neuronal networks correlated with processes involved in lexical access and their time course (eg,[Holcomb, Ph. J., Grainger J. and O'Rourke, T.(2002). An Electrophysiological Study of the Effects of Orthographic Neighborhood Size on Printed Word Perception, J. of Cogn. Neurosci. 14 938–950; Binder, JR, McKiernan, KA, Parsons, ME, Westbury, CF, Possing, ET, Kaufman, JN and Buchanan, L.(2003).
Zusammenfassung. Vor zehn Jahren wurde eine Reform der deutschen Orthografie beschlossen. Anhand ... more Zusammenfassung. Vor zehn Jahren wurde eine Reform der deutschen Orthografie beschlossen. Anhand einer Blickbewegungsstudie, an der sowohl erwachsene Leser als auch Kinder teilnahmen, sollte untersucht werden, welche Auswirkungen die Rechtschreibreform auf das Lesen hatte. Die Ergebnisse liefern erste Hinweise dafür, dass die Rechtschreibreform die Worterkennung insgesamt nicht beeinträchtigt hat.
The present study investigated the nature of the inhibitory syllable frequency effect, recently r... more The present study investigated the nature of the inhibitory syllable frequency effect, recently reported for normal readers, in a German-speaking dyslexic patient. The reading impairment was characterized as a severe deficit in naming single letters or words in the presence of spared lexical processing of visual word forms.
Many neurocognitive studies investigated the neural correlates of visual word recognition, some o... more Many neurocognitive studies investigated the neural correlates of visual word recognition, some of which manipulated the orthographic neighborhood density of words and nonwords believed to influence the activation of orthographically similar representations in a hypothetical mental lexicon. Previous neuroimaging research failed to find evidence for such global lexical activity associated with neighborhood density. Rather, effects were interpreted to reflect semantic or domain general processing. The present fMRI study revealed effects of lexicality, orthographic neighborhood density and a lexicality by orthographic neighborhood density interaction in a silent reading task. For the first time we found greater activity for words and nonwords with a high number of neighbors. We propose that this activity in the dorsomedial prefrontal cortex reflects activation of orthographically similar codes in verbal working memory thus providing evidence for global lexical activity as the basis of the neighborhood density effect. The interaction of lexicality by neighborhood density in the ventromedial prefrontal cortex showed lower activity in response to words with a high number compared to nonwords with a high number of neighbors. In the light of these results the facilitatory effect for words and inhibitory effect for nonwords with many neighbors observed in previous studies can be understood as being due to the operation of a fast-guess mechanism for words and a temporal deadline mechanism for nonwords as predicted by models of visual word recognition. Furthermore, we propose that the lexicality effect with higher activity for words compared to nonwords in inferior parietal and middle temporal cortex reflects the operation of an identification mechanism based on local lexico-semantic activity.
Uploads
Papers by Art Jacobs