Brain Research 1664 (2017) 25–36 Contents lists available at ScienceDirect Brain Research journal homepage: www.elsevier.com/locate/bres Research report Brain activity associated with selective attention, divided attention and distraction Emma Salo a,b,⇑, Viljami Salmela a,b, Juha Salmi a,b,c, Jussi Numminen d, Kimmo Alho a,b a Department of Psychology and Logopedics, Faculty of Medicine, University of Helsinki, Helsinki, Finland Advanced Magnetic Imaging Centre, Aalto Neuroimaging, Aalto University School of Science and Technology, Espoo, Finland c Faculty of Arts, Psychology and Theology, Åbo Akademi University, Turku, Finland d Helsinki Medical Imaging Centre, Helsinki University Hospital, Helsinki, Finland b a r t i c l e i n f o Article history: Received 2 September 2016 Received in revised form 21 February 2017 Accepted 22 March 2017 Available online 28 March 2017 Keywords: Auditory Visual Divided attention fMRI Novel distractors a b s t r a c t Top-down controlled selective or divided attention to sounds and visual objects, as well as bottom-up triggered attention to auditory and visual distractors, has been widely investigated. However, no study has systematically compared brain activations related to all these types of attention. To this end, we used functional magnetic resonance imaging (fMRI) to measure brain activity in participants performing a tone pitch or a foveal grating orientation discrimination task, or both, distracted by novel sounds not sharing frequencies with the tones or by extrafoveal visual textures. To force focusing of attention to tones or gratings, or both, task difficulty was kept constantly high with an adaptive staircase method. A whole brain analysis of variance (ANOVA) revealed fronto-parietal attention networks for both selective auditory and visual attention. A subsequent conjunction analysis indicated partial overlaps of these networks. However, like some previous studies, the present results also suggest segregation of prefrontal areas involved in the control of auditory and visual attention. The ANOVA also suggested, and another conjunction analysis confirmed, an additional activity enhancement in the left middle frontal gyrus related to divided attention supporting the role of this area in top-down integration of dual task performance. Distractors expectedly disrupted task performance. However, contrary to our expectations, activations specifically related to the distractors were found only in the auditory and visual cortices. This suggests gating of the distractors from further processing perhaps due to strictly focused attention in the current demanding discrimination tasks. Ó 2017 Elsevier B.V. All rights reserved. 1. Introduction Our ability to focus on the task at hand is a key element in efficient information processing (Pashler, 1997). Such selective attention is associated with enhanced activity in the sensory and higher level cerebro-cortical areas receiving and processing the attended input, as well in brain areas involved in control of direction of attention (Corbetta and Shulman, 2002; Näätänen, 1990). However, our attention is easily distracted by novel events or changes in the stimulus environment. Task-irrelevant changes even in an unattended sensory modality easily catch our attention and elicit activity both in the sensory cortical areas of the unattended modality, as well as in frontal and parietal regions involved in involuntary attention to distractors and in their evaluation (Alho et al., ⇑ Corresponding author at: Department of Psychology and Logopedics, Faculty of Medicine, PO Box 9, FI 00014 University of Helsinki, Finland. E-mail address: emma.salo@helsinki.fi (E. Salo). http://dx.doi.org/10.1016/j.brainres.2017.03.021 0006-8993/Ó 2017 Elsevier B.V. All rights reserved. 2015; Corbetta and Shulman, 2002; Deouell and Knight, 2009; Lavie, 2005; Näätänen, 1990; Opitz et al., 2002; Rinne et al., 2005; Salmi et al., 2009). Studies in the visual modality (e.g., Corbetta and Shulman, 2002) suggest that stimulus-driven attention activates ventral attention areas, such as posterior parts of the inferior and middle temporal gyri (ITG and MTG, respectively), temporoparietal junction (TPJ) and posterior parts of the inferior and medial frontal gyri (IFG and MFG, respectively). Goal-directed or top-down controlled attention, in turn, activates more dorsal attention areas, such as the intraparietal sulcus (IPS), superior parietal lobule (SPL) and frontal eye fields (FEF). However, the distinction between networks for top-down controlled and stimulus-driven attention may not be as clear as originally proposed. Functional magnetic resonance imaging (fMRI) studies on both auditory and visual attention have shown that stimulus-driven and top-down controlled attention may activate overlapping fronto-parietal cortical areas (Alho et al., 2015; Peelen et al., 2004; Salmi et al., 2009; Serences and Yantis, 2007). 26 E. Salo et al. / Brain Research 1664 (2017) 25–36 It has been shown, that selective attention to one modality during presentation of asynchronous streams of auditory and visual objects enhances activity in the sensory cortex of the attended modality and, in some cases, decreases activity in the sensory cortex of the unattended modality (Johnson and Zatorre, 2006; Laurienti et al., 2002; Mittag et al., 2013; Salo et al., 2013; Serences et al., 2005; Shomstein and Yantis, 2004). In contrast, when auditory and visual stimuli are delivered synchronously, attention-related activity enhancements are seen even in the sensory cortex of the modality to be ignored (Degerman et al., 2007). This is presumably due to unintentional integration of synchronous auditory and visual events into single objects and ‘‘spreading” of attention to the modality to be ignored (see also Busse et al., 2005). Moreover, when attention is intentionally divided between audition and vision, attention-related activity enhancements in the sensory cortices may be lower than during auditory or visual selective attention, due to limited processing resources (Johnson and Zatorre, 2006; Salo et al., 2015). Furthermore, divided attention comprises of several other cognitive processes that recruit cortical areas associated with both ventral and dorsal attention networks. Previous studies have shown, for example, that divided attention is associated with activations in prefrontal cortical areas presumably involved in coordination of dual tasking (Johnson and Zatorre, 2006; Johnson et al., 2007; Moisala et al., 2015; Schubert and Szameitat, 2003; Stelzel et al., 2006; Salo et al., 2015). Previous studies on attention have mainly focused on auditory or visual modality or on one or two aspects of attention. For example, bottom-up capture of attention by novel distractors has been studied in experimental designs requiring selective attention (e.g., Escera et al., 1998; Rees et al., 2001; Wood et al., 2006), but how distractors affect divided attention is less well known. In addition, the networks of bottom-up triggered and top-down controlled attention are usually compared within either auditory or visual modality (e.g., Peelen et al., 2004; Salmi et al., 2009). Yet, there is evidence suggesting that different subregions in frontoparietal areas might be involved in auditory and visual attention (Alho et al., 2015; Braga et al., 2013; Michalka et al., 2015; Salmi et al., 2007, 2009). Since cross-study comparison of brain activations associated with different aspects of attention is complicated by interindividual variation and differences in experimental procedures, our present fMRI study investigated brain activations associated with selective and divided attention to auditory or visual target stimuli, as well as activations elicited by auditory and visual distractors capturing attention in a bottom-up manner. Thus, our aim was to systematically compare all these aspects of attention in the same experiment, which to our knowledge has not been done in any previous study. In all conditions of the present study, the participants were presented with synchronous tone-grating pairs, as well as occasional auditory or visual novel distractors (Fig. 1a). In selective and divided attention conditions, the participants performed 1-back tone pitch or grating orientation discrimination tasks separately or simultaneously, respectively (Fig. 1b). On 1/6 of the trials, an auditory novel distractor and on 1/6 of the trials a visual novel distractor concurred with the task relevant stimuli, however, only one distractor could occur at a time. Thus, the distractors occurred either in the attended or unattended modality. The change in tone pitch and grating orientation was controlled with an adaptive staircase method to keep all conditions comparable in terms of task difficulty. Based on previous studies, we expected to find enhanced activity in the prefrontal and parietal cortex associated with control of attention. In addition, we expected to observe activity enhancements in the sensory cortices of the attended modality during selective attention, and smaller activity enhancements in these areas during divided attention, due to limited processing resources (Johnson and Zatorre, 2006; Salo et al., 2015). However, due to synchronous presentation of auditory and visual stimuli, we expected that activity enhancements associated with selective attention might spread even to the sensory cortices of the modality to be ignored (cf. Busse et al., 2005; Degerman et al., 2007). We were especially interested to see whether or not auditory and visual selective attention are associated with activity enhancements in the same prefrontal and parietal areas, and whether the left dorsolateral prefrontal cortex, implicated in controlling dual tasking (Johnson and Zatorre, 2006; Salo et al., 2015), and superior parietal areas, implicated in intermodal attention switching (Shomstein and Yantis, 2004), would show activity enhancements during divided attention. Furthermore, we expected that the novel distractors would be associated with large activation patterns in the sensory cortical areas, as well as in the ventral parietal and frontal areas associated with stimulus-driven attention (Alho et al., 2015; Corbetta and Shulman, 2002; Salmi et al., 2009). 2. Results 2.1. Task performance To compare discrimination accuracy between different attention conditions, the auditory and visual mean discrimination thresholds (THs) between Attention Modes (selective vs. divided attention) were compared using Bonferroni corrected pairwise comparisons (see 4.4. Analysis of behavioral data for details). THs were lower during selective than divided auditory attention t (14) = 5.607, p < 0.001, and lower during selective than divided visual attention t(14) = 5.101, p < 0.001 (Fig. 1c). To compare reaction times (RTs; only correct responses included) between different trial types, a repeated measures analysis of variance (ANOVA) with factors Attention Mode (selective vs. divided), Modality (auditory vs. visual) and Distractor (auditory vs. visual vs. no distractor) was conducted. This ANOVA revealed significant main effects of Attention Mode F(1,14) = 89.33, p < 0.001, Modality F(1,14) = 35.48, p < 0.001 and Distractor F(2,28) = 11.15, p < 0.001 (in all ANOVAs reported here, the degrees of freedom were Greenhouse–Geisser corrected when needed, but the original degrees are reported with the correction term e and the corrected p; see 4.4 Analysis of behavioral data). As seen in Fig. 1c, the RTs were longer during divided than selective attention, longer to the tones than gratings, and, in most cases, longer on trials with a distractor than for trials with no distractor. There was also a significant Attention Mode  Modality interaction F(1,14) = 5.13, p < 0.05, showing that the RTs to gratings differed more between divided attention and selective visual attention than RTs to tones between divided attention and selective auditory attention (Fig. 1c). The significant Modality  Distractor interaction F(2,28) = 15.83, p < 0.001 showed that auditory RTs were prolonged more by auditory (intramodal) than visual (crossmodal) distractors, and analogously, visual RTs were prolonged more by visual (intramodal) than auditory (crossmodal) distractors (Fig. 1c). Finally, Attention Mode  Modality  Distractor interaction F(2,28) = 4.15, p < 0.05 revealed that during divided attention, RTs to tones were prolonged by occurrence of an auditory or visual distractor, whereas RTs to gratings were prolonged by a concurrent visual distractor. In the control task (where the participants responded by pressing a self-chosen button to all tone-grating pairs), the mean RTs were shorter than during the attention conditions. For control task RTs, a one-way ANOVA indicated significant differences between the trial types F(2,28) = 10.75, p < 0.001, the RTs being longest for trials with visual distractor (mean ± SEM: 645 ± 59 ms), intermediate for trials with an auditory distractor E. Salo et al. / Brain Research 1664 (2017) 25–36 27 Fig. 1. Experimental design and task performance. a) The participants were presented with a stream of synchronous tones and gratings that varied in their pitch and orientation, respectively. On 1/6 of trials, an auditory novel distractor, and on 1/6 of trials, a visual novel distractor concurred together with the tone-grating pair. The auditory distractors were spectrally complex novel sounds, such as bell rings, and the visual distractors were complex colored textures. b) During selective auditory attention the participants were required to indicate with an appropriate button press whether the tone pitch was higher or lower than on the previous trial. During visual attention, they were required to indicate with an appropriate button press whether the grating had turned clockwise or counterclockwise with respect to the previous trial. During divided attention, they were required to indicate with an appropriate button press, in which modality and to which direction stimulus change with respect to previous trial occurred (during divided attention change was restricted to one modality at a time). In a control task, they were required to respond with any button press to all tone-grating pairs. c) Mean discrimination thresholds for tone frequency and grating orientation during selective (Sel) and divided (Div) attention and mean reaction times (RTs; includes only hit responses) for each trial type during selective and divided attention: selective auditory attention (Sel Aud), selective visual attention (Sel Vis), auditory RTs during divided attention (Div Aud) and visual RTs during divided attention (Div Vis). Error bars indicate SEMs. (605 ± 59 ms) and lowest for trials with no distractor (589 ± 56 ms). Taken together, the behavioral results show clear dual task interference, that is, performing two tasks simultaneously both prolonged RTs and reduced discrimination accuracy. Moreover, the prolonged RTs to distracted trials show that the distractors indeed caught the participants’ attention in a bottom-up manner, especially when a distractor occurred in the attended modality. 2.2. Brain activity related to auditory and visual attention We had a total of 12 trial types in the experiment, defined by attention condition (selective auditory attention, selective visual attention, divided attention, control) and the type or presence of distractor (auditory distractor, visual distractor, no distractor). Differences in brain activity during different attention conditions were analysed using a whole brain 2  2  3 repeated measures ANOVA with factors Auditory Attention (auditory attention ‘‘on”, auditory attention ‘‘off”), Visual Attention (visual attention ‘‘on”, visual attention ‘‘off”) and Distractor (auditory distractor, visual distractor, distractor ‘‘off”; see 4.6. Analysis of the fMRI data for further details). Thus, the factor Auditory Attention included selective auditory attention and divided attention conditions, factor Visual Attention included the visual attention and divided attention conditions, and the factor Distractor had three levels, namely, auditory distractor, visual distractor or no distractor. As seen in Fig. 2, the whole brain ANOVA revealed significant F(1,154) > 11.25, voxel-wise p < 0.001) main effects of Auditory Attention and Visual Attention. The main effect of Auditory Attention was seen in the activity of IFG and MFG and in inferior parietal areas bilaterally, and in the left hemisphere superior/middle temporal gyrus (STG/MTG), posterior planum temporale, precentral gyrus and SPL (Fig. 2a, Table 1). The main effect of Visual Attention was seen in the activity bilaterally in the posterior superior frontal gyrus (SFG)/FEF, precentral gyrus and SPL, superior lateral occipital cortex and in the left IFG (Fig. 2b, Table 1). A conjunction analysis of Auditory Attention and Visual Attention revealed significant activity (voxel-wise height threshold t = 3.79, cluster-level p (FWE) < 0.05, cluster size > 50) bilaterally in the precentral gyrus and SPL, extending to superior lateral occipital cortices (Fig. 2c, Table 2; for details of the conjunction analysis, see 4.6. Analysis of the fMRI data). In the medial areas, conjunction of Auditory and Visual Attention was associated with significant activity in the posterior SFG. In short, the results suggest different brain networks for auditory and visual attention. Specifically, different frontal areas were activated during auditory and visual attention, while in parietal areas the activations were partly overlapping. In addition, both Auditory and Visual Attention were associated with enhanced activity in partly overlapping precentral areas, presumably due to different motor responses given to auditory and visual tasks. 2.3. Brain activity related to interaction of auditory and visual attention and divided attention The whole brain ANOVA revealed also significant interactions F (1,154) > 11.25, voxel-wise p < 0.001, cluster-level p(uncorrected) < 0.05) of Auditory and Visual Attention (see 4.6. Analysis of the fMRI data for details) in the left MFG, in the left lateral SPL and in a bilateral medial SPL area extending to superior occipital cortices and in the right ventromedial prefrontal cortex (VMPC) and anterior paracingulate gyrus (Fig. 3a, Table 1; note that the MFG interaction did not survive cluster level p(FWE) < 0.05 correction). 28 E. Salo et al. / Brain Research 1664 (2017) 25–36 Fig. 2. Activity related to auditory and visual attention. Significant (F = 11.25, voxel-wise p < 0.001, cluster size > 50) main effects of a) Auditory Attention and b) Visual Attention from a repeated measures ANOVA with factors Auditory Attention, Visual Attention and Distractor. c) Areas showing significant activity (voxel-wise height threshold t = 3.79, cluster-level p(FWE) < 0.05, cluster size > 50) associated with both Auditory Attention and Visual Attention according to conjunction analysis. Additional post-hoc tests were conducted in the four clusters of voxels showing a significant Auditory Attention  Visual Attention interaction to reveal differences between the conditions contributing to the interaction. For these post-hoc tests, in each voxel cluster the mean percent signal changes in each condition in relation to a baseline period (i.e., stimulus-free periods within and between the tasks blocks; for more details, see 4.6. Analysis of the fMRI data) was calculated. Then differences in these signal changes between the tasks (i.e., selective auditory attention, selective visual attention, divided attention and control task) were tested with Bonferroni corrected pairwise comparisons (see 4.7. Post-hoc and region of interest analysis for further details). These post-hoc tests indicated significant differences between the conditions only in two of these voxel clusters, namely, in the left MFG and right VMPC. In the left MFG, the mean percent signal change was significantly lower during selective auditory attention than during divided attention t (14) = 4.729, p < 0.001, and significantly lower during selective visual attention than during divided attention t(14) = 5.104, p < 0.001 (Fig. 3b). For the right VMPC, the mean percent signal change was significantly higher during selective visual attention than during divided attention t(14) = 3.36, p < 0.05. The activity tended to be higher during selective auditory attention than during divided attention, but this difference did not reach significance, t (14) = 2.815, p < 0.084. The lower activity in the right VMPC during divided attention in relation to other tasks could imply that the current VMPC activity reflected the functioning of a default mode network (e.g., Fox et al., 2005). The activity in the left medial/lateral SPL area was lower during the tasks than during the baseline. The highest activity was observed for the control task, and the lowest activity for auditory and visual selective attention, but according to post-hoc tests the differences between the conditions were non-significant. In the right medial SPL, the highest activity was observed for divided attention and the lowest activity for the auditory and visual selective attention, but again the differences between the conditions were non-significant. To study further brain activity related to divided attention, we separately contrasted brain activity during divided attention with activity during selective auditory attention and activity during selective visual attention. These contrasts were then subjected to conjunction analysis to reveal significant activity enhancements related specifically to divided attention (see 4.6. Analysis of the fMRI data). As seen in Fig. 4 (see also Table 2), these analyses revealed that activity was significantly higher (voxel-wise height threshold t = 3.79, cluster-level p(FWE) < 0.05, cluster size > 50) during divided attention than during both auditory and visual selective attention in the left MFG, precentral gyrus (not visible in Fig. 4) and supplementary motor cortex. 2.4. Brain activity related to distractors The whole brain ANOVA showed also significant main effects of Distractor F(1,154) > 11.25, voxel-wise p < 0.001) bilaterally in the auditory and visual cortices (Table 1). There were no significant interactions between Distractor and the other factors. Since the main effects of Distractor were likely to be modality specific and due to auditory and visual distractors, respectively, direct auditory distractor vs. no distractor and visual distractor vs. no distractor contrasts were conducted. As seen in Fig. 5, significant (voxelwise height threshold t = 3.79, cluster-level p(FWE) < 0.05, cluster size > 50) activity enhancements associated with auditory (Fig. 5a) and visual (Fig. 5b) distractors in comparison with trials with no distractor showed large activation patterns in the auditory and visual sensory cortices, respectively (Table 2). Separate auditory and visual localizer data were used to define ROIs in the auditory and visual cortex, respectively (see 4.7. Post-hoc and region of interest analysis). The mean percent signal changes in these ROIs during all 12 trial types in relation E. Salo et al. / Brain Research 1664 (2017) 25–36 29 Table 1 Main effects of Auditory Attention, Visual Attention and Distractor, and interaction of Auditory and Visual Attention. MNI coordinates and F-value global maxima within activation clusters of significant (F = 11.25, cluster-level p(uncorr.) < 0.05, cluster size > 50) activations. Note that all but two activation clusters (with bold) survive p(FWE) < 0.05 correction. The brain areas are labelled according to Harvard-Oxford Cortical Structural Atlas and Cerebellar Atlas in MNI152 space. F-value Brain region p uncorr. CLUSTER-level p(FWE) Cluster size 79.64 51.71 40.56 38.45 36.03 35.63 34.17 32.25 31.29 29.29 28.54 23.67 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.003 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.061 0.001 0.000 0.000 6661 2689 2436 894 378 1012 2034 418 90 188 294 308 24 6 8 32 60 46 48 30 42 28 32 32 12 52 20 46 12 76 20 24 22 42 28 78 60 4 44 28 14 32 26 4 24 10 56 50 80.58 57.36 55.86 51.96 51.26 50.95 35.09 30.53 30.43 26.88 26.83 23.80 22.49 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.002 0.000 0.001 0.001 0.001 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.038 0.000 0.012 0.018 0.015 767 2113 905 488 2646 622 1288 302 101 601 131 120 125 24 36 28 8 20 44 8 32 40 48 30 40 8 8 44 0 12 64 4 44 58 30 52 20 68 74 56 56 52 52 58 30 6 30 12 10 4 30 26 224.67 89.33 72.47 14.95 13.99 10.61 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.004 0.000 0.006 19,966 4647 4136 139 303 133 10 56 54 40 34 0 74 24 26 28 10 42 6 12 8 14 28 38 33.76 20.10 19.16 18.98 0.000 0.003 0.017 0.001 0.000 0.058 0.317 0.023 1233 91 52 1113 20 28 40 8 76 76 18 48 54 56 30 14 MNI coordinates x y z Main effect of Auditory Attention Left Left Right Right Left Left Right Left Right Left Left Right Precentral gyrus Paracingulate gyrus Posterior cingulate gyrus Cerebellum, VI Posterior middle temporal gyrus Superior lateral occipital cortex Pars opercularis Insular cortex Parietal operculum cortex Lingual gyrus Middle frontal gyrus Superior lateral occipital cortex Main effect of Visual Attention Left Left Right Left Right Right Left Right Left Right Left Left Left Superior frontal gyrus Superior parietal lobule Middle frontal gyrus Paracingulate gyrus Superior lateral occipital cortex Precentral gyrus Paracingulate gyrus Cerebellum, VI Frontal orbital cortex Posterior inferior temporal gyrus Insular cortex Cerebellum, Crus I Cerebellum, VI Main effect of Distractor Right Right Left Left Right Lingual gyrus Planum temporale Planum temporale Pars triangularis Precentral gyrus Paracingulate gyrus Interaction of Auditory and Visual Attention Left Right Left Right Superior lateral occipital cortex Superior lateral occipital cortex Middle frontal gyrus Paracingulate gyrus to a baseline period are shown in Fig. 6. For the auditory cortex, an ANOVA with factors Auditory Attention, Visual Attention and Distractor revealed only a main effect of Distractor F(2,28) = 39.05, p < 0.001. As seen in Fig. 6, the activity in the auditory cortex was significantly higher for trials with an auditory distractor than for trials with no distractor and trials with visual distractor. A similar ANOVA for the visual cortex revealed also only a main effect of Distractor F(2,28) = 41.85, p < 0.001, the activity being higher for trials with a visual distractor than for trials with no distractor or for trials with an auditory distractor (Fig. 6). Thus both auditory and visual distractors elicited significant activity enhancements in the respective modality-specific cortical areas. Post-hoc tests between no distractor trials during auditory selective attention, visual selective attention, divided attention and control tasks indicated no significant activity differences in the auditory cortex. In the visual cortex, the activity was significantly higher during selective visual attention than during selective auditory attention t(14) = 3.479, p < 0.01, divided attention t (14) = 3.887, p < 0.01, and control task t(14) = 3.041, p < 0.01. 3. Discussion 3.1. Auditory and visual attention Auditory attention was associated with bilateral activity in IFG/MFG implicated in top-down controlled, goal-directed auditory attention also in previous studies (e.g., Alain et al., 2008; Alho et al., 2015; Degerman et al., 2006; Salmi et al., 2007; Salo et al., 2013). In addition, auditory attention was associated with activity in the left temporal areas and bilateral IPL, presumably due to auditory processing (e.g., Alain et al., 2001; Arnott et al., 2004; Arnott and Alain, 2011; Bushara et al., 1999; Salo et al., 2013; Warren and Griffiths, 2003; Weeks et al., 1999). Visual attention to spatial orientation of the gratings was associated with widespread activity in SPL, bilaterally. The activity in SPL has previously been associated with goal-directed, top-down controlled visual attention and spatial processing (Bushara et al., 1999; Corbetta and Shulman, 2002; Corbetta et al., 1993; Giesbrecht et al., 2003; Le et al., 1998; Salmi et al., 2007; Salo et al., 2013; Yantis et al., 2002). In addition, visual attention was E. Salo et al. / Brain Research 1664 (2017) 25–36 30 Table 2 Activity during auditory and visual attention as revealed by conjuction analysis, activity during divided attention in relation to selective auditory and visual attention and activity related to auditory and visual distractors, respectively. MNI coordinates and t-value global maxima within activation clusters of significant (t = 3.79, cluster-level p(FWE) < 0.05, cluster size > 50) activations. The brain areas are labelled according to Harvard-Oxford Cortical Structural Atlas and Cerebellar Atlas in MNI152 space. t-value Brain region Cluster-level p(FWE) Cluster size MNI coordinates x y z Conjunction of Auditory Attention vs. No Auditory Attention and Visual Attention vs. No Visual Attention Right Right Right Left Right Left Left Right Right Right Left Superior frontal gyrus Precentral gyrus Precentral gyrus Precentral gyrus Superior parietal lobule Superior parietal lobule Cerebellum, VI Superior lateral occipital cortex Superior lateral occipital cortex Cerebellum, VI Superior lateral occipital cortex 9.07 7.05 6.43 6.42 6.23 5.99 5.98 5.46 5.32 5.32 4.72 0.001 0.000 0.000 0.000 0.000 0.000 0.016 0.001 0.000 0.003 0.078 122 200 285 159 159 314 83 122 145 109 60 8 28 52 28 28 34 32 30 16 32 8 20 2 6 16 48 44 64 76 68 60 68 52 46 34 50 42 54 26 26 58 30 60 Conjunction of Divided Attention vs. Auditory Selective Attention and Divided Attention vs. Visual Selective Attention Left Left Middle frontal gyrus Superior frontal gyrus 6.52 6.46 0.003 0.000 100 133 44 8 18 10 36 56 8.71 7.66 7.61 0.000 0.000 0.032 2641 2521 75 54 58 18 6 38 36 4 10 26 13.88 5.51 0.000 0.012 16,341 97 28 2 36 56 14 22 Auditory distractor vs. No distractor Right Left Right Planum polare Posterior superior temporal gyrus Frontal pole Visual distractor vs. No distractor Right Right Posterior parahippocampal gyrus Frontal pole Fig. 3. Activity related to interaction of auditory and visual attention. a) Voxels showing significant (F = 11.25, voxel-wise p < 0.001, cluster size > 50) Auditory Attention  Visual Attention interaction in a repeated measures ANOVA. Note that the left MFG activity does not survive cluster level p(FWE) < 0.05 correction (see Table 1 for details). b) Mean percent signal changes during different tasks in relation to a baseline period in the left MFG and right VMPC areas, selective auditory attention (Sel Aud), selective visual attention (Sel Vis), divided attention (Div), and the Control task (Ctrl). Error bars indicate SEMs. Conditions differing significantly from each other are indicated with asterisks (* p < 0.05, *** p < 0.001). associated with bilateral activity in the occipital cortex likely due to attention towards task relevant visual features (Corbetta et al., 1990; Heinze et al., 1994; Kastner and Ungerleider, 2000; Reynolds and Chelazzi, 2004; Schwartz et al., 2005). Incongruently with our previous study (Salo et al., 2013), we did not observe markedly enhanced activity in the auditory and visual cortices during selective auditory and visual attention, respectively. However, there were some activity differences in the audi- tory and visual cortex ROIs between selective auditory and selective visual attention even for trials with no distractors (the white bars in Fig. 6). The additional post-hoc tests indicated that for trials with no distractors, the activity in the visual cortex was significantly higher during selective visual attention than during selective auditory attention, while the opposite difference was not significant in the auditory cortex. The lack of marked effects of selective attention in sensory cortices was not, however, E. Salo et al. / Brain Research 1664 (2017) 25–36 31 Fig. 4. Activity in brain areas that according to a conjuction analysis showed significant (voxel-wise height threshold t = 3.79, cluster-level p(FWE) < 0.05, cluster size > 50) activity enhancements during divided attention in relation to both auditory selective attention and visual selective attention. Fig. 5. Brain activity associated with auditory and visual distractors. Areas showing significant (voxel-wise height threshold t = 3.79, cluster-level p(FWE) < 0.05, cluster size > 50) activity enhancements for trials with a) an auditory distractor or b) a visual distractor in relation to trials with no distractor (data combined across all task conditions). Fig. 6. Mean percent signal changes in the auditory and visual cortex regions of interest (ROIs) for different trial types in relation to a baseline period during selective auditory attention (Sel Aud), selective visual attention (Sel Vis), divided attention (Div) and the Control task (Ctrl). Error bars indicate SEMs. unexpected. Since the tones and gratings were delivered synchronously, it is possible that they were perceptually bound into audiovisual objects leading to attention-related activity enhancements in both sensory cortices during selective attention (Busse et al., 2005; Degerman et al., 2007). Alternatively, even without audiovisual binding, due to synchronous auditory and visual stimulation, attention-related activity enhancements might have spread unintentionally to the modality to be ignored. Finally, it is also possible, that the slow stimulus presentation rate resulted in attenuated effects of selective attention in the sensory areas (Ozus et al., 2001; Rinne et al., 2005). According to conjunction analysis, both auditory and visual attention were associated with bilateral activity enhancements in the SPL, previously implicated in visual attention and spatial processing (e.g., Corbetta and Shulman, 2002; Salo et al., 2013; Yantis et al., 2002) as already discussed above, but also in supramodal top-down controlled orienting of attention (e.g., Corbetta and Shulman, 2002; Salmi et al., 2007; Shomstein and Yantis, 32 E. Salo et al. / Brain Research 1664 (2017) 25–36 2004). The fact that SPL activity was enhanced both during selective attention to the spatial orientation of gratings and during selective attention to a non-spatial feature of the tones (i.e., their pitch), suggests frequent orienting of attention during both auditory and visual selective attention tasks perhaps due to the slow presentation rate of tone-grating pairs complicating constant maintenance of attention on the current tasks. Conjunction analysis indicated also bilateral activity in the precentral areas and cerebellum and in posterior parts of the superior frontal gyri for both auditory and visual attention. These activations were presumably related to selection and execution of the motor responses (Le et al., 1998; Meier et al., 2008; note that the simple reaction time task in the control condition did not require response selection). the current VMPC activity might be best explained by deactivation of the default mode network during the goal-directed tasks. It should be noted that during divided attention the taskrelevant change could occur only in one modality at a time. Thus, the participants might have ceased to process the stimulus in one modality after noticing a target change in the other modality. However, this would have decreased, rather than increased, activity related to divided attention. In addition, due to the continuously controlled task difficulty, the discrimination tasks in both modalities were performed at the threshold. Therefore, the participants were likely forced to process the stimuli of both modalities on every trial. 3.3. Brain activity related to distractors 3.2. Divided attention As expected, we observed increments in auditory and visual discrimination thresholds and reaction times during divided attention in comparison with selective attention indicating that the divided attention task was more demanding than either of the selective attention tasks. The whole brain ANOVA revealed a significant interaction of Auditory and Visual Attention in the left MFG, implicated in divided attention (Corbetta and Shulman, 2002; Johnson et al., 2007; Johnson and Zatorre, 2006; Moisala et al., 2015; Salo et al., 2015; Schubert and Szameitat, 2003; Stelzel et al., 2006). Moreover, the post-hoc analyses in the left MFG area showed that activity was significantly higher during divided attention than during the selective attention tasks in the left MFG. However, the interaction in the MFG area did not survive cluster level p(FWE) < 0.05 correction. Furthermore, it should be noted that the present interaction of Auditory Attention and Visual Attention is not analogous to divided attention as also activity during the Control task may have contributed to this interaction. Therefore, direct contrasts between divided attention vs. selective auditory attention and divided attention vs. selective visual attention were subjected to a conjunction analysis to reveal activity specifically associated with divided attention (Salo et al., 2015). This analysis revealed a significant activity enhancement associated with divided attention in the same left MFG area that showed significant Auditory Attention  Visual Attention interaction. Nevertheless, one might ask, whether this area is involved in supramodal processing rather than divided attention. However, the conjunction analysis for the factors Auditory Attention and Visual Attention revealed no conjunction of activations in this area. This suggests that with regard to attention, this MFG area is involved in cognitive processes needed specifically during divided attention (Corbetta and Shulman, 2002; Johnson et al., 2007; Johnson and Zatorre, 2006; Moisala et al., 2015; Salo et al., 2015; Schubert and Szameitat, 2003; Stelzel et al., 2006). The interaction of Auditory Attention and Visual Attention was also associated with activity in the right VMPC, in the same area that was activated by Auditory Attention and Visual Attention separately. Previous studies have connected the VMPC with the default mode network, activated during rest and deactivated during goal-directed tasks (e.g., Fox et al., 2005; Simpson et al., 2001; McKiernan et al., 2003). In addition, during goal-directed tasks, the VMPC activity has been shown to decrease with increasing task difficulty (Simpson et al., 2001; McKiernan et al., 2003). Similarly, in the current study, the ROI analysis showed that in the right VMPC, activity was lower during divided attention than during the other tasks, albeit only the difference between divided attention and visual selective attention was significant. The VMPC and adjacent areas have also been implicated in processing of distracting stimuli (Corbetta and Shulman, 2002; Corbetta et al., 2008; Salmi et al., 2009; Salo et al., 2015). Yet, in the current study, we found no significant VMPC activity related to distractors. Thus, Prolonged reaction times on distracted trials indicated, as expected, that the distractors indeed caught participants’ attention. Previous studies have suggested that such stimulus-driven attention is associated with enhanced activity in the inferior parietal and frontal cortices (e.g., Alho et al., 2015; Corbetta and Shulman, 2002). However, when auditory and visual novel distractors are presented simultaneously with task relevant stimuli, their effects on brain activity are mostly found in and near the auditory and visual cortices, respectively. The enhanced activity elicited in the auditory and visual cortices by trials with auditory and visual distractors, respectively, is presumably due to activation of new afferent elements, as well as preattentive change detection processes in these cortical areas typically activated by infrequent ‘‘oddball” stimuli occurring among repetitive ‘‘standard” stimuli (see, e.g., Alho et al., 2014; Kimura et al., 2011; Näätänen et al., 2007). However, it was unexpected that significant activity enhancements were not observed in frontal and parietal areas for the distractors that, according to prolonged reaction times, caught involuntary attention in a bottom-up manner (cf. Alho et al., 2015; Corbetta and Shulman, 2002; Opitz et al., 2002; Salmi et al., 2009). Perhaps strictly focused attention to the target events due to the demanding discrimination tasks performed at the threshold prevented further fronto-parietal processing of the distractors (cf. Lavie, 2005). It should be noted, however, that in the present study, the auditory distractors included only frequencies outside the attended frequency range of tones and the visual distractors occurred outside the attended foveal location. Therefore rejection of these distractors from further processing may have been easier than rejection of distractors sharing the attended frequency range or spatial location. 3.4. Conclusions The present study investigated the effects of different attention modes (selective or divided), attended modality (auditory or visual) and distractors (auditory or visual) in the same experiment, not done previously. In addition, unlike in many previous related studies, the task difficulty during different attention conditions was controlled. First, the present results indicate that both auditory and visual selective attention are associated activity enhancements in frontal and parietal cortical areas presumably involved in top-down control of attention, and perhaps also further processing of attended stimuli (e.g., Salo et al., 2013). However, the activations associated with auditory and visual selective attention were observed largely in different frontal areas, while the parietal activations overlapped significantly. In accordance with previous studies, this suggests that different subregions in frontal areas, but partly overlapping parietal regions are activated during goal-directed auditory and visual attention (Michalka et al., 2015; Salmi et al., 2007). E. Salo et al. / Brain Research 1664 (2017) 25–36 Second, an activity enhancement specifically associated with divided attention was observed in the left MFG in accordance with previous studies suggesting participation of this area in top-down control of division of attention between audition and vision or integration of performance in auditory and visual tasks during dual tasking (Johnson and Zatorre, 2006; Salo et al., 2015). Third, the auditory and visual distractors elicited activity enhancements only in the auditory and visual cortex, respectively. This was presumably due to strictly focused attention to the nearthreshold target events, which may have prevented further frontoparietal processing of the distractors that occurred outside this focus (i.e., outside the frequency range of target tones and the spatial location of target gratings). 4. Experimental Procedure 4.1. Participants Participants (N = 15, 8 female) were native Finnish speakers, between 19 and 37 years of age (mean 26 years). All participants were right handed, had normal hearing, normal or corrected-tonormal vision, and no history of psychiatric or neurological illnesses (all self reported). An informed written consent was obtained from each participant before the experiment. The experimental protocol was approved by the Ethical Review Board in the Humanities and Social and Behavioural Sciences, University of Helsinki. All participants had participated in a previous EEG study, during which similar tasks were performed. Thus, the participants were well trained to perform the experimental tasks. 4.2. Stimuli The participants were presented with synchronous sinewave tones and sinewave gratings. In addition, on some trials an auditory or visual novel distractor concurred with the task relevant stimuli. All stimuli had a duration of 300 ms. The frequency of sinewave tones varied between 600 and 1800 Hz in steps depending on task performance (see 4.3. Procedure). The auditory distractors were spectrally complex novel sounds, such as rising scales of beeps, car honks etc. They were high- and low-pass filtered with cut-offs at 200 and 7000 Hz, respectively. In addition, to avoid acoustic masking of the tones, the novel sounds were notch-filtered at 1000 Hz with a two octave wide filter. The sinewave tones and distractors had 10 ms rise and fall times. The sounds were filtered with AudacityÒ audio editor (version 2.0.6, http://www.audacityteam.org). The maximum intensity of all sounds was 80 dB SPL. The grayscale sinewave gratings varied in orientation between 0 and 360° in steps depending on task performance (see 4.3. Procedure). The radius of the circular grating was 1.5° and the grating had a spatial frequency of 2 c/deg. The visual distractors were complex colored textures subtending 16°  24° around the grating. The root-mean-square contrast (standard deviation of luminance divided by mean luminance) of the textures was 0.3. A circular 6° area was cut off from center of textures to avoid overlapping with the sinewave gratings. To study activity in the sensory cortices, separate auditory and visual functional localizers were presented. The auditory functional localizer was a 2000 ms stream of twenty 100 ms sinewave tones (including 10 ms rise and fall times of each tone), delivered at an intensity of 80 dB SPL, with its frequency varying randomly between 600 and 1800 Hz. The visual functional localizer was a circular checkerboard (diameter 3°) flickering at 8 Hz for 2000 ms at the center of the screen. 33 The sounds were delivered binaurally using Sensimetrics S14 insert earphones (Sensimetrics Corporation, Malden, MA, USA). Noise from the scanner was attenuated by the earplugs and viscoelastic mattresses inside and around the headcoil and under the participant. The visual stimuli were presented on a gray background and projected to a mirror fixed to the head coil. 4.3. Procedure Task relevant stimuli were sinewave tones and sinewave gratings presented synchronously. The stimulus pairs had a constant onset-to-onset interval of 1800 ms. In addition, on 1/6 of trials, a task-irrelevant auditory novel distractor and on 1/6 of trials, a task-irrelevant visual novel distractor concurred with the taskrelevant stimuli, however, only one distractor could occur at a time. Each novel distractor was used only once during the experiment. The experiment consisted of 1-back auditory and visual discrimination tasks involving a stream of synchronous tones and gratings (Fig. 1a). The auditory discrimination task was to indicate by a right hand button press whether the pitch of a sinewave tone was higher or lower than the preceding sinewave tone by pressing an up or down button, respectively (Fig. 1b). The visual discrimination task was to indicate by a right hand button press whether the orientation of the grating rotated clockwise or counterclockwise in comparison with the preceding grating by pressing a right or left button, respectively. During divided attention, the participants were required to indicate, with the same four buttons, in which modality and to which direction the change occurred. There was also a control condition, during which the participants were asked to press any response button to all tone-grating pairs. During divided attention and control tasks change in tones and gratings was restrained to occur only in one modality, while during selective attention a change could occur also in the unattended modality. The difficulty of discrimination tasks was maintained approximately constant by keeping the rate of correct responses at 70.7% with an adaptive staircase method based on trials without distractors. The initial change in pitch and orientation was 0.1 octaves and 15°, respectively. The amount of pitch and orientation change was increased after an incorrect response and decreased after two subsequent correct responses by 0.01 octaves or 3°, respectively. The maximum change between consecutive trials was limited to 0.5 octaves and 90°. The experiment was conducted in three functional runs. Each run contained one block of each task in random order. A block consisted of 60 trials, of which 10 trials included an auditory distractor and 10 trials a visual distractor. (In addition, each run contained blocks of unimodal auditory and visual trials. To form a balanced whole brain analysis of variance, the unimodal trials, including distractors only in one modality, were excluded from the present analysis.) A written task instruction was presented on the screen for 2 s before the onset of each block and after each block a written feedback was given for 3 s. All instructions and feedbacks consisted of one line of text. The instructions were ‘‘Do the Auditory/ Visual/ Control task” or ‘‘Do the Auditory and Visual tasks” and the feedback was either ‘‘You are doing well. Keep up the good work” (over 50% hit rate) or ‘‘You made some errors. Try to concentrate more” (50% hit rate or lower). The instructions and feedback were in Finnish. In addition to instructions and feedback, there were short, stimulus-free breaks within and after each block (25 s in total per block). During the stimulus-free periods, no experimental stimuli were presented and participants waited for the experiment to continue. The tones and gratings were created, and stimulus timing and randomization was controlled with the Presentation software (version 14.9, Neurobehavioral Systems, Berkeley, CA, USA). 34 E. Salo et al. / Brain Research 1664 (2017) 25–36 After the actual experiment, 2 blocks with the auditory localizer stimuli and 2 blocks with visual localizer stimuli were delivered. The participants were instructed to fixate their gaze at the center of the screen without any task instruction. The localizer stimuli were presented for 2000 ms, followed by a 500 ms silent period with no stimuli. Each localizer stimulus was presented 10 times within a block. The total duration of each block was 25 s. The blocks were delivered in random order. 4.4. Analysis of behavioral data The discrimination thresholds for pitch and orientation in selective and divided attention conditions were calculated by averaging reversal points (the first two reversal points were excluded; see 4.3. Procedure for more details). Statistical significance of differences in auditory and visual mean THs between the different Attention Modes (selective vs. divided attention) were tested with Bonferroni corrected pairwise comparisons. The mean reaction times (RTs) of hits, determined as correct responses given 300–1800 ms after target stimulus onset, were calculated separately within each task for trials with auditory, visual or no distractor. Hit RTs were analysed using a 2  2  3 repeated measures ANOVA with factors Attention Mode (selective vs. divided), Modality (auditory vs. visual) and Distractor (auditory vs. visual vs. no distractor). In the ANOVAs, the degrees of freedom were Greenhouse-Geisser corrected when needed. However, the original degrees of freedom will be reported below together with the corrected p-value. The reported correction term e implicates corrections. 4.5. fMRI data acquisition and initial analysis Functional brain imaging was carried out with a 3.0 T MAGNETOM Skyra whole body scanner (Siemens Healthcare, Erlangen, Germany) using a 30 channel head coil. The functional gradientecho echo planar images (EPI) were acquired with an imaging area consisting of 33 oblique axial slices (TR 1900 ms, TE 32 ms, flip angle 75°, voxel matrix 64  64, field of view 20 cm, slice thickness 3.0 mm, between slice gap 1 mm, in-plane resolution 3.1 mm  3.1 mm  3.0 mm). Jittered image acquisition was used (i.e., stimulus presentation and data acquisition were not time locked). The functional scanning was divided in three 12 min task runs resulting in 3  383 functional volumes. A high-resolution anatomical images (voxel matrix 176  256  256, in-plane resolution 1 mm  1 mm  1 mm) were acquired after two functional runs to offer the participants a short break from the task. Finally, at the end of the session, a functional auditory localizer and visual localizer of 40 volumes each were acquired from 13 participants (in two participants, no localizer data could be collected because this would have exceeded the scanning time limit in these participants). Between the runs there were short breaks during which participants remained in the scanner and were instructed not to move their heads. 4.6. Analysis of the fMRI data The data were analysed with Statistical Parametric Mapping (SPM12) toolbox (Wellcome Department of Cognitive Neurology, London, UK) and with custom Matlab scripts. In order to allow for initial stabilization of the fMRI signal, the first four volumes of each run were excluded from analysis. The data were motion corrected, spatially smoothed (Gaussian kernel of 6 mm fullwidth half-maximum), high-pass filtered (cutoff 1/128 Hz), and then transformed into a standard space (MNI152; Montreal Neurological Institute). The hemodynamic response was modeled using canonical hemodynamic response function (mean lag 6 s, SD 1 s). Based on timing information recorded during the experiment, 12 trial types were defined with regard to Condition (selective auditory attention, selective visual attention, divided attention, control) and Distractor (auditory distractor, visual distractor, no distractor). In addition, 6 nuisance regressors for motion (movement and rotation along three orthogonal axes) and 1 nuisance regressor for instructions (2 s periods of written instruction before the next task) and feedback (3 s periods of written feedback presented after a task) were added to the model. The remaining data were defined as baseline (i.e., periods without experimental stimuli within and between the tasks). Several contrasts were specified to study brain activity related to different tasks and different distractor types. Differences in brain activity between trial types were analysed using a whole brain 2  2  3 repeated measures ANOVA with factors Auditory Attention (auditory attention ‘‘on”, auditory attention ‘‘off”), Visual Attention (visual attention ‘‘on”, visual attention ‘‘off”) and Distractor (auditory distractor, visual distractor, and distractor ‘‘off”). Thus, the factor Auditory Attention included selective auditory attention and divided attention conditions, factor Visual Attention included the visual attention and divided attention conditions, and the factor Distractor had three levels, namely, auditory distractor, visual distractor or no distractor. In addition, to compare activations associated with auditory and visual attention and to study brain activity associated with divided attention, additional conjunction analyses were performed. To compare activations associated with auditory and visual attention, similar contrasts as in the whole brain ANOVA were formed (for auditory attention, selective auditory attention and divided attention were contrasted with selective visual attention and the control task, and for visual attention, selective visual attention and divided attention were contrasted with selective auditory attention and control task). These contrasts were then subjected to conjunction analysis to reveal significant activation enhancements associated with both auditory and visual attention. To study activity related to divided attention in a similar way as in our previous fMRI study (Salo et al., 2015), we separately contrasted brain activity during the divided attention with activity during the selective auditory and selective visual attention. The resulting statistic images were then entered into conjunction analysis to reveal significant activation enhancements specific to divided attention. All statistical images were thresholded using voxel-wise significance threshold p < 0.001 and cluster size > 50. The significant activations, corrected at cluster-level according to familywise error correction (FWE) based on the random field theory, are reported in Tables 1 and 2. 4.7. Post-hoc and region of interest analysis For both post-hoc and regions of interest (ROI) analyses, the mean percent signal changes within a particular voxel cluster during different conditions were calculated in relation to data collected during stimulus-free baseline periods within and between the task blocks. To study activity in the sensory cortices, the data of 13 participants from the localizer runs were analysed with separate general linear model (GLM). Two stimulus-related variables and contrasts for auditory tones and visual checkerboards were defined. Statistical images were thresholded using voxel-wise significance threshold p < 0.0001, corresponding a t-value of 5.2. The group level data were used to define ROIs in the auditory and visual cortices used for all 15 participants. The mean percent signal E. Salo et al. / Brain Research 1664 (2017) 25–36 changes in these ROIs during all 12 trial types were collected. To study the effects of attention and distractor type on these activations, a 2  2  3 repeated measures ANOVA similar to that for whole brain analysis (i.e., with factors Auditory Attention, Visual Attention and Distractor) was conducted. In addition, to study further effects of divided attention, posthoc tests were performed in voxel clusters showing significant interaction of Auditory Attention and Visual Attention in the whole brain ANOVA. These voxel clusters were defined in Freesurfer software to cover the voxels showing a significant Auditory Attention  Visual Attention interaction. For each of these voxel clusters, we calculated the mean percent signal changes for each of the four tasks in relation to the baseline. The activity in these areas during the different tasks was then tested with Bonferroni corrected pairwise comparisons. Acknowledgements This research was supported by the Finnish Cultural Foundation and the Academy of Finland (grant #260054). References Alain, C., Arnott, S.R., Hevenor, S., Graham, S., Grady, C., 2001. ‘‘What” and ‘‘where” in the human auditory system. Proc. Natl. Acad. Sci. U. S. A. 98, 12301–12306. Alain, C., He, Y., Grady, C., 2008. The contribution of the inferior parietal lobe to auditory spatial working memory. J. Cogn. Neurosci. 20, 285–295. Alho, K., Herron, T.J., Rinne, T., Woods, D.L., 2014. Stimulus-dependent activations and attention-related modulations in the auditory cortex: a meta-analysis of fMRI studies. Hearing Res. 307, 29–41. Alho, K., Salmi, J., Koistinen, K., Salonen, O., Rinne, T., 2015. Top-down controlled and bottom-up triggered orienting of auditory attention to pitch activate overlapping brain networks. Brain Res. 1626, 136–145. Arnott, S.R., Alain, C., 2011. The auditory dorsal pathway: orienting vision. Neurosci. Biobehav. Rev. 35, 2162–2173. Arnott, S.R., Binns, M.A., Grady, C., Alain, C., 2004. Assessing the auditory dualpathway model in humans. NeuroImage 22, 401–408. Braga, R.M., Wilson, L.R., Sharp, D.J., Wise, R.J., Leech, R., 2013. Separable networks for top-down attention to auditory non-spatial and visuospatial modalities. NeuroImage 74, 77–86. Bushara, K.O., Weeks, R.A., Ishii, K., Catalan, M.J., Tian, B., Rauschecker, J.P., Hallet, M., 1999. Modality-specific frontal and parietal areas for auditory and visual spatial localization in humans. Nat. Neurosci. 2, 759–766. Busse, L., Roberts, K.C., Crist, R.E., Weissman, D.H., Woldorff, M.G., 2005. The spread of attention across modalities and space in a multisensory object. Proc. Natl. Acad. Sci. U. S. A. 102, 18751–18756. Corbetta, M., Shulman, G.L., 2002. Control of goal-directed and stimulus-driven attention in the brain. Nat. Rev. Neurosci. 3, 201–215. Corbetta, M., Miezin, F.M., Dobmeyer, S., Shulman, G.L., Petersen, S.E., 1990. Attentional modulation of neural processing of shape color and velocity in humans. Science 248, 1556–1559. Corbetta, M., Miezin, F.M., Shulman, G.L., Petersen, S.E., 1993. A PET study of visuospatial attention. J. Neurosci. 13, 1202–1226. Corbetta, M., Patel, G., Shulman, G.L., 2008. The reorienting system of the human brain: from environment to theory of mind. Neuron 58, 306–324. Degerman, A., Rinne, T., Salmi, J., Salonen, O., Alho, K., 2006. Selective attention to sound location or pitch studied with fMRI. Brain Res. 1077, 123–134. Degerman, A., Rinne, T., Pekkola, J., Autti, T., Jääskeläinen, I., Sams, M., Alho, K., 2007. Human brain activity associated with audiovisual perception and attention. NeuroImage 34, 1683–1691. Deouell, L.Y., Knight, R.T., 2009. Executive function and higher-order cognition: EEG studies. In: Squire, L.R. (Ed.), Encyclopedia of Neuroscience. Academic Press, Oxford, UK, pp. 105–109. Escera, C., Alho, K., Winkler, I., Näätänen, R., 1998. Neural mechanisms of involuntary attention to acoustic novelty and change. J. Cogn. Neurosci. 10, 590–604. Fox, M.D., Snyder, A.Z., Vincent, J.L., Corbetta, M., Van Essen, D.C., Raichle, M.E., 2005. The human brain is intrinsically organized into dynamic, anticorrelated functional networks. Proc. Natl. Acad. Sci. U. S. A. 102, 9673–9678. Giesbrecht, B., Woldorff, M.G., Song, A.W., Mangun, G.R., 2003. Neural mechanisms of top-down control during spatial and feature attention. NeuroImage 19, 496– 512. Heinze, H.J., Mangun, G.R., Burchert, W., Hinrichs, H., Scholz, M., Münte, T.F., Gös, A., Scherg, M., Johannes, S., Hundeshagen, H., Gazzaniga, M.S., Hillyard, S.A., 1994. Combined spatial and temporal imaging of brain activity during visual selective attention in humans. Nature 372, 543–546. 35 Johnson, J.A., Zatorre, R.J., 2006. Neural substrates for dividing and focusing attention between simultaneous auditory and visual events. NeuroImage 31, 1673–1681. Johnson, J.A., Strafella, A.P., Zatorre, R.J., 2007. The role of the dorsolateral prefrontal cortex in bimodal divided attention: two transcranial magnetic stimulation studies. J. Cogn. Neurosci. 19, 907–920. Kastner, S., Ungerleider, L.G., 2000. Mechanisms of visual attention in the human cortex. Annu. Rev. Neurosci. 23, 315–341. Kimura, M., Schröger, E., Czigler, I., 2011. Visual mismatch negativity and its importance in visual cognitive sciences. NeuroReport 22, 669–673. Laurienti, P.J., Burdette, J.H., Wallace, M.T., Yen, Y.F., Field, A.S., Stein, B.E., 2002. Deactivation of sensory-specific cortex by cross-modal stimuli. J. Cogn. Neurosci. 14, 420–429. Lavie, N., 2005. Selective attention under load. Trends Cogn. Sci. 9, 75–82. Le, T.H., Pardo, J.V., Hu, X., 1998. 4T-fMRI study of nonspatial shifting of selective attention: cerebellar and parietal contributions. J. Neurophysiol. 79, 1535– 1548. McKiernan, K.A., Kaufman, J.N., Kucera-Thompson, J., Binder, J.R., 2003. A parametric manipulation of factors affecting task-induced deactivation in functional neuroimaging. J. Cogn. Neurosci. 15, 394–408. Meier, J.D., Alfalo, T.N., Kastner, S., Graziano, M.S.A., 2008. Complex organization of human primary motor cortex: a high-resolution fMRI study. J. Neurophysiol. 100, 1800–1812. Michalka, S.W., Kong, L., Rosen, M.L., Shinn-Cunningham, B.G., Somers, D.C., 2015. Short-term memory for space and time flexibly recruit complementary sensorybiased frontal lobe attention networks. Neuron 87, 882–892. Mittag, M., Inauri, K., Huovilainen, T., Leminen, M., Salo, E., Rinne, T., Kujala, T., Alho, K., 2013. Attention effects on the processing of task-relevant and task-irrelevant speech sounds and letters. Front. Neurosci. 231, 1–15. Moisala, M., Salmela, V., Salo, E., Carlson, S., Vuontela, V., Salonen, O., Alho, K., 2015. Brain activity during divided and selective attention to auditory and visual sentence comprehension tasks. Front. Hum. Neurosci. 9, 86. Näätänen, R., 1990. The role of attention in auditory information processing as revealed by event-related potentials and other brain measures of cognition function. Behav. Brain Sci. 13, 201–288. Näätänen, R., Paavilainen, P., Rinne, T., Alho, K., 2007. The mismatch negativity (MMN) in basic research of central auditory processing: a review. Clin. Neurophysiol. 118, 2544–2590. Opitz, B., Rinne, T., Mecklinger, A., von Cramon, D.Y., Schröger, E., 2002. Differential contribution of frontal and temporal cortices to auditory change detection: fMRI and ERP results. NeuroImage 15, 167–174. Ozus, B., Liu, H., Chen, L., Iyer, M., Fox, P., Gao, J., 2001. Rate dependence of human visual cortical response due to brief stimulation: an event-related fMRI study. Magn. Reson. Imaging 19, 21–25. Pashler, H.E., 1997. The Psychology of Attention. MIT Press, Boston, MA, USA. Peelen, M.V., Heslenfeld, D.J., Theeuwes, J., 2004. Endogenous and exogenous attention shifts are mediated by the same large-scale neural network. NeuroImage 22, 822–830. Rees, G., Frith, C., Lavie, N., 2001. Processing of irrelevant visual motion during performance of an auditory attention task. Neuropsychologia 39, 937–949. Reynolds, J.H., Chelazzi, L., 2004. Attentional modulation of visual processing. Annu. Rev. Neurosci. 27, 611–647. Rinne, T., Pekkola, J., Degerman, A., Autti, T., Jääskeläinen, I.P., Sams, M., Alho, K., 2005. Modulation of auditory cortex activation by sound presentation rate and attention. Human Brain Mapp. 26, 94–99. Salmi, J., Rinne, T., Degerman, A., Salonen, O., Alho, K., 2007. Orienting and maintenance of spatial attention in audition and vision: multimodal and modality-specific brain activations. Brain Struct. Funct. 212, 181–194. Salmi, J., Rinne, T., Koistinen, S., Salonen, O., Alho, K., 2009. Brain networks of bottom-up triggered and top-down controlled shifting of auditory attention. Brain Res. 1286, 155–164. Salo, E., Rinne, T., Salonen, O., Alho, K., 2013. Brain activity during auditory and visual phonological, spatial and simple discrimination tasks. Brain Res. 1496, 55–69. Salo, E., Rinne, T., Salonen, O., Alho, K., 2015. Brain activations during bimodal dual tasks depend on the nature and combination of component tasks. Front Hum. Neurosci. 9, 102. Schubert, T., Szameitat, A.J., 2003. Functional neuroanatomy of interference in overlapping dual tasks: an fMRI study. Cogn. Brain Res. 17, 733–746. Schwartz, S., Vuilleumier, P., Hutton, C., Maravita, A., Dolan, R., Driver, J., 2005. Attentional load and sensory competition in human vision: modulation of fMRI responses by load at fixation during task-irrelevant stimulation in the peripheral visual field. Cereb. Cortex 15, 770–786. Serences, J., Yantis, S., 2007. Spatially selective representations of voluntary and stimulus-driven attentional priority in human occipital parietal and frontal cortex. Cereb. Cortex 17, 284–293. Serences, J., Shomstein, S., Leber, A.B., Egeth, H., Yantis, S., 2005. Coordination of voluntary and stimulus-driven attentional control in human cortex. Psychol. Sci. 16, 114–122. Shomstein, S., Yantis, S., 2004. Control of attention shifts between vision and audition in human cortex. J. Neurosci. 24, 10702–10706. Simpson, J.R., Snyder, A.Z., Gusnard, D.A., Raichle, M.E., 2001. Emotion-induced changes in human medial prefrontal cortex: I. During cognitive task performance. Proc. Natl. Acad. Sci. U. S. A. 98, 683–687. 36 E. Salo et al. / Brain Research 1664 (2017) 25–36 Stelzel, C., Schumacher, E.H., Schubert, T., D’Esposito, M., 2006. The neural effect of stimulus-response modality compatibility on dual-task performance: an fMRI study. Psychol. Res. 70, 514–525. Warren, J.D., Griffiths, T.D., 2003. Distinct mechanisms for processing spatial sequences and pitch sequences in the human auditory brain. J. Neurosci. 23, 5799–5804. Weeks, R.A., Aziz-Sultan, A., Bushara, K.O., Tian, B., Wessinger, C.M., Dang, N., Rauschecker, J.P., Hallet, M., 1999. A PET study of human auditory spatial processing. Neurosci. Lett. 262, 155–158. Wood, J., Chaparro, A., Hickson, L., Thyer, N., Carter, P., Hancock, J., Hoe, A., Le, I., Sahetapy, L., Ybarzabal, F., 2006. The effect of auditory and visual distracters on the useful field of view: implications for the driving task. Invest. Ophthalmol. Vis. Sci. 47, 4646–4650. Yantis, S., Schwarzbach, J., Serences, J.T., Carlson, R.L., Steinmetz, M.A., Pekar, J.J., Courtney, S.M., 2002. Transient neural activity in human parietal cortex during spatial attention shifts. Nat. Neurosci. 5, 995–1002.