Parallels in the structure of speech and music have long fascinated great thinkers from Plato to ... more Parallels in the structure of speech and music have long fascinated great thinkers from Plato to Darwin. The communication of emotion is no exception, as it touches on one of the primary motivations for musical listening – feeling “moved by the music.” Although cues such as pitch height and timing are widely recognized as playing an important role in both domains, their role in music has been less researched than in language. Here I will discuss several of my team’s explorations aimed at providing a musical complement to existing work in speech and linguistics. The plethora of speech corpora provide useful data for examining the “natural” use of acoustic communicate emotion. Unfortunately, the clear contrast between the number of ‘effective speakers’ and the number of ‘effective composers’ complicates efforts to fully explore parallels in structural cues between these domains. Numerous studies based on manipulations of simplified musical stimuli such as newly composed single line melodies suggest important parallels between the communication of emotion in speech and music. Melodies transposed higher in pitch sound “happier” and melodies played at slower tempi sound “sadder” – paralleling the use of these cues in speech. However, it is not clear whether these simplified approaches capture the nuanced ways in which great composers employ such cues in their writing. Here I will discuss my team’s research on this important issue, which mixes the techniques of acoustical analysis, psychophysical testing, data visualization, and empirical musicology to provide a diverse exploration of musical emotion complementing and extending speech studies. In doing so my team aims to deepen our understanding of mechanisms shared between these ubiquitous human activities, providing insights useful to linguistics, musicians, psychologists, and cognitive scientists alike.
Journal of the Acoustical Society of America, Apr 1, 2014
Here, we extend research indicating that a sound's amplitude envelope (changes in intensity o... more Here, we extend research indicating that a sound's amplitude envelope (changes in intensity over time) affects subjective duration (Grassi and Darwin, 2006). Specifically, we explore whether amplitude envelope affects the strategies underlying duration perception by comparing “flat” (abrupt onset, sustain, abrupt offset) and “percussive” (abrupt onset followed by exponential decay) envelopes. Participants performed a two alternative forced choice task: judging which of two tones sounded longer. Trials were divided into blocks organized by envelope: (a1) uniform flat-flat, (a2) uniform percussive-percussive, and (b) mixed (uniform, percussive-flat, and flat-percussive). This was designed to either permit (a) or prohibit (b) envelope-specific listening strategies. Block order was counterbalanced across participants. An analysis of the first block (a between-subjects comparison of uniform and mixed block performance) indicated that performance on flat-flat trials was not significantly different for the two block types, whereas percussive-percussive trial performance was significantly worse for the mixed block. This suggests that participants optimally employ an envelope-specific strategy for the uniform block (a) when able to predict envelope, and a generalized strategy in the mixed block (b) when unable. Interestingly, there was no performance advantage for the uniform block when presented second, suggesting that contextual order effects may affect auditory duration perception.
A sound’s decay conveys useful information to listeners, such as the materials as well as the for... more A sound’s decay conveys useful information to listeners, such as the materials as well as the force involved in the impact event (Gygi, Kidd, & Watson, 2004). Yet its role is often overlooked in auditory perception experiments employing trapezoidally-shaped “flat” amplitude envelopes. My team has documented that time varying sounds can lead to considerably different understandings of perceptual organization in tasks ranging from audio-visual integration (Schutz, 2009) and duration assessment (Vallet, Shore, & Schutz, 2014) to memory recall (Schutz, Stefanucci, Baum & Roth, 2017). To establish a baseline understanding of the range of sounds used in auditory perception research, we analyzed 215 experiments from 111 articles published the Journal of the Acoustical Society of America (JASA) using methodology similar to my team’s previous survey of the journal Music Perception (Vaisberg & Schutz, 2014). Here we found 78% of auditory perception stimuli exhibited “flat” amplitude envelopes, with clicks/click trains accounting for an additional 7.8%. Only 2.8% exhibited the kinds of complex changes in amplitude found in natural sounds. Although the remaining 11.2% used sounds with some time varying amplitude information, this variation lacked any real-world referent (i.e., amplitude modulated tones, “pyramid shaped” tones with matched rise/fall times, etc.). Therefore less than 3% of the sounds exhibited the kinds of complex changes in amplitude found in (and informative about) natural sounds. Moreover 86% of stimuli used in our survey exhibited no amplitude variation beyond abrupt onsets/offsets. This distribution is broadly consistent with my team’s surveys of other journals, indicating the under-assessment of sounds with complex amplitude variations is wide spread issue within psychological acoustics. I will discuss the implications of this previously undocumented challenge, as well as highlight implications for future research opportunities
Journal of the Acoustical Society of America, Apr 1, 2015
Spatial ventriloquism occurs when a visual event and an auditory event happen simultaneously; loc... more Spatial ventriloquism occurs when a visual event and an auditory event happen simultaneously; location judgments for the sound source become biased toward the location of the visual event. Evidence suggests the brain attributes greater weight to spatial information provided by the visual stimulus because the visual system offers better spatial acuity; when the visual stimulus is deteriorated, the visual bias is reduced. Thus, the brain performs optimal bimodal integration: greater weight is given to the modality which provides more information. The present study aims to determine whether the amplitude envelope of sounds provides spatial localization information to the perceptual system. We used a psychophysical staircase procedure to measure spatial ventriloquism experienced by participants for sounds with percussive, flat, and time-reversed percussive envelopes. We hypothesize that percussive and reverse-percussive sounds provide more information and thus better sound localization acuity than flat sounds, which would result in smaller degrees of spatial ventriloquism for the former than the latter. The results yield insight into the brain’s use of auditory cues in audio-visual integration.
Large, E. W., & Jones, M. R. (1999). The dynamics of attending: How people track time-varying eve... more Large, E. W., & Jones, M. R. (1999). The dynamics of attending: How people track time-varying events. Psychological
Here, we demonstrate that “moving to the beat ” can improve the perception of timing, providing a... more Here, we demonstrate that “moving to the beat ” can improve the perception of timing, providing an intriguing explanation as to why we often move when listening to music. In the first experiment, participants heard a series of isochro-nous beats and identified whether the timing of a final tone after a short silence was consistent with the timing of the preceding sequence. On half of the trials, participants tapped along with the beat, and on half of the trials, they listened without moving. When the final tone occurred later than expected, performance in the movement condition was signif-icantly better than performance in the no-movement condition. Two additional experiments illustrate that this improved per-formance is due to improved timekeeping, rather than to a shift in strategy. This work contributes to a growing literature on sensorimotor integration by demonstrating body movement’s objective improvement in timekeeping, complementing previ-ous explorations involving subjectiv...
Quality care for patients requires effective communication amongst medical teams. Increasingly, c... more Quality care for patients requires effective communication amongst medical teams. Increasingly, communication is required not only between team members themselves, but between members and the medical devices monitoring and managing patient well-being. Most human–computer interfaces use either auditory or visual displays, and despite significant experimentation, they still elicit well-documented concerns. Curiously, few interfaces explore the benefits of multimodal communication, despite extensive documentation of the brain’s sensitivity to multimodal signals. New approaches built on insights from basic audiovisual integration research hold the potential to improve future human–computer interfaces. In particular, recent discoveries regarding the acoustic property of amplitude envelope illustrate that it can enhance audiovisual integration while also lowering annoyance. Here, we share key insights from recent research with the potential to inform applications related to human–computer...
Parallels in the structure of speech and music have long fascinated great thinkers from Plato to ... more Parallels in the structure of speech and music have long fascinated great thinkers from Plato to Darwin. The communication of emotion is no exception, as it touches on one of the primary motivations for musical listening – feeling “moved by the music.” Although cues such as pitch height and timing are widely recognized as playing an important role in both domains, their role in music has been less researched than in language. Here I will discuss several of my team’s explorations aimed at providing a musical complement to existing work in speech and linguistics. The plethora of speech corpora provide useful data for examining the “natural” use of acoustic communicate emotion. Unfortunately, the clear contrast between the number of ‘effective speakers’ and the number of ‘effective composers’ complicates efforts to fully explore parallels in structural cues between these domains. Numerous studies based on manipulations of simplified musical stimuli such as newly composed single line melodies suggest important parallels between the communication of emotion in speech and music. Melodies transposed higher in pitch sound “happier” and melodies played at slower tempi sound “sadder” – paralleling the use of these cues in speech. However, it is not clear whether these simplified approaches capture the nuanced ways in which great composers employ such cues in their writing. Here I will discuss my team’s research on this important issue, which mixes the techniques of acoustical analysis, psychophysical testing, data visualization, and empirical musicology to provide a diverse exploration of musical emotion complementing and extending speech studies. In doing so my team aims to deepen our understanding of mechanisms shared between these ubiquitous human activities, providing insights useful to linguistics, musicians, psychologists, and cognitive scientists alike.
Journal of the Acoustical Society of America, Apr 1, 2014
Here, we extend research indicating that a sound's amplitude envelope (changes in intensity o... more Here, we extend research indicating that a sound's amplitude envelope (changes in intensity over time) affects subjective duration (Grassi and Darwin, 2006). Specifically, we explore whether amplitude envelope affects the strategies underlying duration perception by comparing “flat” (abrupt onset, sustain, abrupt offset) and “percussive” (abrupt onset followed by exponential decay) envelopes. Participants performed a two alternative forced choice task: judging which of two tones sounded longer. Trials were divided into blocks organized by envelope: (a1) uniform flat-flat, (a2) uniform percussive-percussive, and (b) mixed (uniform, percussive-flat, and flat-percussive). This was designed to either permit (a) or prohibit (b) envelope-specific listening strategies. Block order was counterbalanced across participants. An analysis of the first block (a between-subjects comparison of uniform and mixed block performance) indicated that performance on flat-flat trials was not significantly different for the two block types, whereas percussive-percussive trial performance was significantly worse for the mixed block. This suggests that participants optimally employ an envelope-specific strategy for the uniform block (a) when able to predict envelope, and a generalized strategy in the mixed block (b) when unable. Interestingly, there was no performance advantage for the uniform block when presented second, suggesting that contextual order effects may affect auditory duration perception.
A sound’s decay conveys useful information to listeners, such as the materials as well as the for... more A sound’s decay conveys useful information to listeners, such as the materials as well as the force involved in the impact event (Gygi, Kidd, & Watson, 2004). Yet its role is often overlooked in auditory perception experiments employing trapezoidally-shaped “flat” amplitude envelopes. My team has documented that time varying sounds can lead to considerably different understandings of perceptual organization in tasks ranging from audio-visual integration (Schutz, 2009) and duration assessment (Vallet, Shore, & Schutz, 2014) to memory recall (Schutz, Stefanucci, Baum & Roth, 2017). To establish a baseline understanding of the range of sounds used in auditory perception research, we analyzed 215 experiments from 111 articles published the Journal of the Acoustical Society of America (JASA) using methodology similar to my team’s previous survey of the journal Music Perception (Vaisberg & Schutz, 2014). Here we found 78% of auditory perception stimuli exhibited “flat” amplitude envelopes, with clicks/click trains accounting for an additional 7.8%. Only 2.8% exhibited the kinds of complex changes in amplitude found in natural sounds. Although the remaining 11.2% used sounds with some time varying amplitude information, this variation lacked any real-world referent (i.e., amplitude modulated tones, “pyramid shaped” tones with matched rise/fall times, etc.). Therefore less than 3% of the sounds exhibited the kinds of complex changes in amplitude found in (and informative about) natural sounds. Moreover 86% of stimuli used in our survey exhibited no amplitude variation beyond abrupt onsets/offsets. This distribution is broadly consistent with my team’s surveys of other journals, indicating the under-assessment of sounds with complex amplitude variations is wide spread issue within psychological acoustics. I will discuss the implications of this previously undocumented challenge, as well as highlight implications for future research opportunities
Journal of the Acoustical Society of America, Apr 1, 2015
Spatial ventriloquism occurs when a visual event and an auditory event happen simultaneously; loc... more Spatial ventriloquism occurs when a visual event and an auditory event happen simultaneously; location judgments for the sound source become biased toward the location of the visual event. Evidence suggests the brain attributes greater weight to spatial information provided by the visual stimulus because the visual system offers better spatial acuity; when the visual stimulus is deteriorated, the visual bias is reduced. Thus, the brain performs optimal bimodal integration: greater weight is given to the modality which provides more information. The present study aims to determine whether the amplitude envelope of sounds provides spatial localization information to the perceptual system. We used a psychophysical staircase procedure to measure spatial ventriloquism experienced by participants for sounds with percussive, flat, and time-reversed percussive envelopes. We hypothesize that percussive and reverse-percussive sounds provide more information and thus better sound localization acuity than flat sounds, which would result in smaller degrees of spatial ventriloquism for the former than the latter. The results yield insight into the brain’s use of auditory cues in audio-visual integration.
Large, E. W., & Jones, M. R. (1999). The dynamics of attending: How people track time-varying eve... more Large, E. W., & Jones, M. R. (1999). The dynamics of attending: How people track time-varying events. Psychological
Here, we demonstrate that “moving to the beat ” can improve the perception of timing, providing a... more Here, we demonstrate that “moving to the beat ” can improve the perception of timing, providing an intriguing explanation as to why we often move when listening to music. In the first experiment, participants heard a series of isochro-nous beats and identified whether the timing of a final tone after a short silence was consistent with the timing of the preceding sequence. On half of the trials, participants tapped along with the beat, and on half of the trials, they listened without moving. When the final tone occurred later than expected, performance in the movement condition was signif-icantly better than performance in the no-movement condition. Two additional experiments illustrate that this improved per-formance is due to improved timekeeping, rather than to a shift in strategy. This work contributes to a growing literature on sensorimotor integration by demonstrating body movement’s objective improvement in timekeeping, complementing previ-ous explorations involving subjectiv...
Quality care for patients requires effective communication amongst medical teams. Increasingly, c... more Quality care for patients requires effective communication amongst medical teams. Increasingly, communication is required not only between team members themselves, but between members and the medical devices monitoring and managing patient well-being. Most human–computer interfaces use either auditory or visual displays, and despite significant experimentation, they still elicit well-documented concerns. Curiously, few interfaces explore the benefits of multimodal communication, despite extensive documentation of the brain’s sensitivity to multimodal signals. New approaches built on insights from basic audiovisual integration research hold the potential to improve future human–computer interfaces. In particular, recent discoveries regarding the acoustic property of amplitude envelope illustrate that it can enhance audiovisual integration while also lowering annoyance. Here, we share key insights from recent research with the potential to inform applications related to human–computer...
Uploads
Papers by Michael Schutz