A Way for Deaf and Hard of Hearing People to Enjoy Music by Exploring and Customizing Cross-modal Music Concepts

Alternative sensory channels, such as visual and tactile, have been used to support DHH people with barriers to experiencing music in activities ranging from appreciation to music to creative activities. As summarized in Table 1, tactile feedback has been used in various ways for music appreciation [36, 37, 61, 83], and instrumental performance [58, 78, 84]. However, visual feedback is typically not used alone, but is combined with tactile feedback to provide detailed feedback on the informational aspects of music learning, such as playing [40, 98], singing [101], training [27], and music-making [48, 73, 86], rather than music appreciation.

For DHH people, the tactile sense is often more intuitive and natural used than the visual sense, allowing them to experience music through sensations, such as feeling the vibrations from loudspeakers [11]. However, research on enabling DHH people to appreciate and express music through the sense of vibration has primarily focused on rhythmic elements-driven presentations using vibration [38, 48, 72, 73]. This is due to the limited bandwidth of the tactile vibrations that can be perceived and differentiated by humans compared to the auditory bandwidth [5, 8, 93]. Therefore, to convey a broader range of musical elements, research is being conducted on whether the pitch or emotion of music can be distinguished through haptic frequency. For example, to convey melody to DHH people, Karam et al. [36, 37] developed an Emoti-Chair that converts audio signals into a tactile channel using a “model human cochlea.” In addition, several studies have examined whether DHH people can distinguish the melody and emotion of music with the vibration of the musical elements. However, the perception and differentiation of frequencies are personal, and some require training to make a clear distinction of music [46, 80].

Graphic-based visualizers that convert musical elements into visuals have been proposed primarily for music education through aural training and music theory. For music training and theory learning, DHH people have been given simplified musical feedback by converting musical elements into visuals or by designing and presenting these elements as music scores. Zhou et al. [101] developed a mobile game with auditory training that maps pitch to the height and color of visual artifacts for children with cochlear implants, and Ling [49] proposed an easy-to-read visual music sheet using color and shape design to help deaf children better comprehend a chord and an interval. Although there are studies that have supported DHH people in music appreciation through visualization, there has yet to be an accepted music-visual mapping convention. Fourney and Fels [17] suggested 18 visual types using shape created and reported the difficulties and limitations involved in music enjoyment among DHH people. As the mapping of music to visual elements primarily aims to convey the informational aspects of music to obtain detailed feedback and intuitive visual notation, its application to the appreciation of music by DHH people is limited [25]. Pouris et al. [74] reported that existing graphic-based visualizers are only practical for music experts who can analyze music visually, indicating that it is ineffective for audiences, such as DHH people, who do not understand music theory or cannot experience music professionally.

Existing research has explored the use of visual and tactile senses to facilitate and enrich experience in musical activities, such as appreciating, playing instruments, and composing both for hearing individuals and for DHH people (Table 1). For hearing professionals, visual and tactile elements have been mainly used to enhance the understanding and analysis of music [4, 9, 13, 21, 24, 29, 47, 54, 79, 85]. Drawing and paintings have been used to enhance the accessibility of music, as well as its easy creation without musical experts, by non-professionals, especially children [12, 96, 97]. Most of these prior studies about substitution system for hearing people focused on identifying mapping relationships between music (e.g., pitch, loudness, rhythm), visual (e.g., color, size, and shape) [9, 24, 29], and tactile (e.g., frequency) [4] elements to enable hearing people to understand and represent music intuitively with the sound [12, 13, 15, 39, 55, 60, 68, 87].

For DHH people, challenges in accessing sound can potentially affect their ability to perceive the clarity of these mapping rules in musical experiences [81]. Graphic visualizations for hearing people were primarily aimed at understanding and modeling the structure of music, so applying these to the music appreciation experience of DHH people may be challenging [17, 29]. Consequently, making DHH people to adapt to the relationships between the senses is necessary [3, 16]. As shown in Table 1, we surveyed previous works to see if their substitution system includes identifying the mapping relationships between music and assistive modalities (exploration) and designing substituted sensory according to personal taste and preferences (customization). More specifically, the systems were categorized according to whether their user was actively or passively involved in the exploration process. We found that most studies do not support the exploration phase as a tool or limitedly support it in a passive way, for example, by allowing the user to be informed about the mapping rules without voluntary participation. On the other hand, to help users with hearing impairments experience playing a musical instrument, the MuSS-Bits [73] provides visual and tactile feedback to actively engage users to explore different sounds, including musical instruments (e.g., drum) and external sounds (e.g., hand clapping). In addition to sound exploration strategies, the authors noted that personalization of tactile vibration is necessary for DHH users to be comfortable with the accepted music. However, it is difficult to see the cross-modal relationship of how exploring the mapping relationships between sensors affects the music appreciation experience. Although prior research suggests that customization phase can enhance the music-making experience [72, 73], current music sensory substitution systems for appreciation are deficient in personalizing musical elements.

Existing studies lack that simultaneously supported an active exploration phase and customization to grasp the mapping relationships between alternative senses. By designing a system that supports active and easy navigation and customization, we aim to further enhance the music appreciation experience of DHH people by replacing the traditional visual and tactile approach with music.

To identify the needs, challenges, and strategies required to design a music-sensory substitution system, we conducted focus group interviews with a total of 24 DHH people, consisting of a group with three participants. To select the people for the interview, we recruited for specific conditions through social welfare facilities, and these conditions include hearing limitation, an interest in music, and experience with visual or tactile-based music-sensory substitution system; however, it was not necessary for participants to have musical training or a high level of musical understanding. The participant demographic is shown in Table A1 in the Appendix (5 males and 19 females; mean age = 35.79 years [SD = 11.62]; and with hearing impairments [13 deaf and 11 hard of hearing]). All DHH participants had previous experience with music-sensory substitution systems, such as visualizers or tactile devices. Semi-structured interviews were conducted mainly on the potentials and challenges of music-sensory substitution systems. The three participants within one group answered questions one-on-one through a sign language interpreter, and the interview lasted an average of 70 min. Each participant was compensated with $15. This study was approved by Gwangju Institute of Science and Technology’s Institutional Review Board (IRB).

Video and audio recordings of all interviews were created with the participants’ consent. Two researchers transcribed and coded all the recordings independently using thematic analysis [6]. They developed a comprehensive codebook and reviewed the coding results with the co-authors to gain further insight into the music–sensory substitution experiences of DHH people.

Based on the interviews, we identified music–sensory substitution systems as an important means of increasing the access of DHH people to music experiences, but there are issues around cross-modal sensory perception and personalization that need to be addressed. To achieve the application of a music–sensory substitution system to the music experience of DHH people, first, a process of adequate awareness and familiarity on the relationships between music-related sensory needs to be implemented. Second, there should be a process of identifying sensory preferences and personalizing them accordingly. In this study, we defined the internalization of musical information through these two processes as conceptualization. Thus, we derived design goals to realize the conceptualization of music–sensory substitution based on the comments of the interviewees and the findings of related studies.

Exploration (Figure 1 a) is designed to help users conceptualize cross-modal mapping rules by observing and representing them while playing he instrument. As shown in Table 1, existing research has been conducted on the intuitive mappings between music and visual and tactile elements. In previous research, visual color, position, and size were mapped to musical elements, whereas tactile was primarily mapped to intensity because frequency requires a learning period. Additionally, our system also follows the intuitive and congruent mapping rules of these studies, using the piano to conceptualize melodic elements (pitch, rhythm, and volume) and the drum to conceptualize rhythmic elements (rhythm, tempo, and beat). Moreover, the system relies on the concept of super-additive processing, a phenomenon in which multimodal senses acting simultaneously produce an effect greater than the sum of their parts, to deliver music to three senses at once: sound, visual and tactile [66, 88, 94, 95]. Our design phase triggered the expectation that multimodality would be more effective than unimodality, as studies have demonstrated that multimodality facilitates the processing of different stimuli, increases detection accuracy, and reduces reaction time to feedback [70]. In this study, visual and tactile sensations were conveyed simultaneously as a user explores cross-modal mapping through drumming and piano playing. When a user clicks the drum image, a dot appears in the middle of the screen, and the haptic device vibrates according to the cross-modal mapping rules. For the piano input, the user can tap a total of 36 piano key buttons (12 scales × 3 octaves), and dots sequentially appear at positions with colors corresponding to the pitch. To support the exploration, the task of playing freely or playing while looking at the sheet of a specific music was included in this part. Free playing allows the user to express visual and tactile senses using an instrument, spontaneously creating a personal interpretation of the musical elements beyond the learned cross-modal mapping rule or basic music theory. With a visual-based music sheet, the user has to play the piano or drum at the right moment just as in a rhythm game, such as Perfect Piano¹. A transparent gray progress bar over the dots moves from left to right for a total of 15 s, indicating when to play. Under these conditions, users must play the piano or drum correctly to match the pitch and rhythm of the given music (i.e., the visual stimuli) with the timing of the progress bar.

Customization (Figure 1 b) is a process of changing the stimuli on the exploration display through cross-modal mapping based on the preference of DHH individuals. In our system, users can create and express the sensory representation—an expression in which musical elements are replaced with visual and tactile senses—to match their preferences. The visual elements that design the melody and beat can be adjusted for size, color, texture, shape, and spacing between pitches. They can also change the sensitivity and intensity with respect to the tactile feedback. Sensitivity corresponds to the least sound that can be ignored when setting the threshold for haptic feedback. If the sensitivity is set to the highest level, tactile feedback occurs even for very small sounds. In contrast, if the sensitivity is set to the lowest level, haptic feedback appears for only loud sounds. Intensity determines how strongly the tool generates the vibrations. Customization enables the users to explore cross-modal mapping rules while selecting and designing multimodal sensory options. Additionally, it provides a process for users to explore and design sensations that work for them, making the experience of alternative senses more intuitive and comfortable.

After conceptualization through exploration and customization, music appreciation allows users to appreciate music with the corresponding multimodal stimuli. The musical elements, such as timbre, pitch, and volume, of the songs analyzed are converted into visual and tactile sensations based on cross-modal mapping and user customization. Users can import their desired mp3 files of any music they want to experience and appreciate the music based on three cross-modal stimuli: sound, visual, and tactile stimuli. Tactile stimuli were rendered using Tactosy for Arms, from bHaptics², by receiving a beat and rhythm signal from our system. This commercial product offers an application programming interface (API) for customizing the intensity and sensitivity of the tactile signal, indicating its suitability for implementing our framework. The appreciation interface can automatically translate any music into visual and tactile sensations to help DHH users identify and feel music, dance videos, and other music-related content.

For this study, we recruited 28 DHH with hearing impairments ranging from hearing loss to deafness, with previous experience music activities, and who had not participated in the previous interviews. There were no restrictions on hearing status, gender, or age, but we excluded people who had no experience or familiarity with computers. Thereafter, they were divided into an experimental group and a control group (i.e., with and without the conceptualization phase, respectively) considering their age, identity, auditory ability, and previous music experience through a preliminary questionnaire (Table 2). The average ages of the participants in the control and experimental groups were 33 (SD = 7.75) and 35.8 years (SD = 8.05), respectively. Both groups included an equal proportion of six deaf people with severe to profound hearing loss (85+ dB), who could recognize nearby loud sounds, and eight hard-of-hearing people who did not have difficulty hearing common sounds after wearing a hearing aid but found it difficult to enjoy music due to sound distortion.

We conducted a counter-balanced, between-subject evaluation to verify if the iterative usage of our system creates an adequate conceptualization of cross-modal mapping and enhances the music appreciation experience. Conceptualization involves the exploration of the mapping relations between visual and tactile elements by playing a virtual instrument, followed by a customization phase in which detailed visual and tactile elements are designed. The users appreciated the music through exploration and personalized sensory substitutions. For music appreciation, we selected 20 music samples of each genre (Pop, R&B, Rock, and Classic) from Jamendo Music, a free music-sharing website, and then selected four music samples that the participants had not experienced through pre-survey [32].

To confirm the satisfaction of the participants with the visual and tactile feedback–based music appreciation, a questionnaire that measures the participants’ subjective experience of music appreciation was designed. Based on the questions from a previous study, we designed the music experience questionnaire (MEQ) to enable sign language interpretation for participants who have difficulty reading [59]. As shown in Table 3, our subjective MEQ includes nine questions in three categories: awareness of changes in musical elements (three items), satisfaction (four items), and emotion (two items) with the music appreciation experience, measured on a 7-point Likert scale. Herein, emotions were not measured on a per-music sample basis, but rather, attention was paid to the overall sensory experience. For example, participants were asked to rate whether they felt more positive (valence) or more excited (arousal) about the overall experience of the alternative sensory music.

The post-interview consisted of open-ended questions divided into the conceptualization phase and the appreciation phase. In the conceptualization phase, questions were asked about whether the exploration and customization processes supported their perception and representation of cross-modal mappings (e.g., “How did you perceive the cross-modal mapping relationships?”, “How did you design the sensations?”). For the appreciation phase, the questions focused on the reasons for positive/negative feelings about the appreciation experience (e.g., “Why was the music appreciation positive?”, “How did the conceptualization phase influence your feelings?”).

Participants in both groups engaged in five sessions on different days [76]. Both groups used the same system, but their sessions were structured differently depending on whether they performed the conceptualization phase. Both groups appreciated music through multimodal senses in the first session for baseline measurement. The type of task differed across groups in the second, third, and fourth sessions, and the tasks focused on usage related to cross-modal conceptualization for music appreciation. The experimental group appreciated music after 20 min of exploration and customization, whereas the control group appreciated music directly through the alternative senses without cross-modal conceptualization. In the fifth session, both groups were requested to repeat the same task from the first session. As DHH people may listen to music with residual hearing during actual music experiences, sound was included for all conditions.

The detailed process for each session in the experimental group was described as follows, with the control group repeating the first session of the experimental group five times:

Before the experiment, participants signed a consent form and completed a demographic questionnaire. In all sessions, we explained the purpose of the experiment and conducted it while seated in a quiet room with a sign language interpreter. During the experiment, the sign language interpreter was asked to interpret everything, including the conversations of users among themselves. Musical stimuli were ordered by counterbalance using a random Latin Square. After each 30s music clip ended, participants were given a cool-down period of 40s in line with the music clip time. After the main experiment, participants participated individually in a post-interview lasting approximately 10 min. It was a semi-structured interview, divided into the phases of conceptualization and appreciation phases, as mentioned above, and was conducted one-on-one with the help of an interpreter who had helped carry out the experiment. All interviews were recorded with the consent of the participants. Each participant was compensated with $20 per session, and this study was approved by Gwangju Institute of Science and Technology’s Institutional Review Board (IRB).

Figure 2 shows the results of the awareness of changes in musical elements and the improvement in satisfaction and emotion during the five sessions under the multimodality condition. The trend in the score differences between sessions differed for the experimental and control groups: data from the experimental group exhibited a positive trajectory, whereas data from the control group displayed a slight decrease. Using the Mann–Whitney U test to compare the data from experimental and control groups, we found statistically significant differences in all categories during the fifth session (Table 4). In particular, pitch (p=0.0001), loudness (p=0.0009), and satisfaction (p=0.0001) were highly statistically significant. This suggests that our approach, aimed at promoting conceptualization through exploration and customization, was successful in improving music appreciation across repeated sessions.

Music appreciation of the experimental group gradually improved over five sessions. However, participants’ subjective overall music experience of exploration and customization activities was rated lower in the second and third sessions than in the first session without any activities. Participants required considerable effort to comprehend the cross-modal mapping relations in a short time and internalize them into a complete musical experience. In the initial sessions, participants indicated a disconnect with the concept and design process of cross-modal mapping. This may have resulted in a slightly diminished musical experience. A more detailed explanation is provided in Section 6.2.

Thus, the experience in the fifth session improved compared to the first session. For all categories of music appreciation, the comparison between the first and fifth sessions demonstrated statistical significance with the Wilcoxon signed-rank test (Table 4). Especially, pitch (p=0.00063) and loudness (p=0.00059) showed a positive and highly statistically significant development in the fifth session. In contrast, for the control group, pitch (p=0.015), valence (p=0.027), arousal (p=0.012), and satisfaction (p=0.046) all showed a statistically significant decrease from the first session to the fifth session, as determined by a Wilcoxon signed-rank test (Table 4). Despite continued exposure, the participants did not appear to internalize the musical elements that were altered by the cross-modal mapping. Wilcoxon signed-rank test results revealed that the awareness of loudness and rhythm did not show statistical significance in the fifth session compared to the first session; even pitch exhibited a statistically significant decrease.

The participants in the experimental group engaged in exploration and developed a conceptual image of the cross-modal mapping of music. In the early stages of their exploration, they revealed that they struggled with accepting the basic concept of music. Additionally, the observation of the control group indicated that struggles in the exploration phase could affect the entire music experience of DHH people with a sour first expression, resulting in restricted access to the system and eventually leading them to lose motivation for the music and further activities. To prevent this, the music-sensory substitution system needs to provide a sufficient and accessible exploration phase in the early stage. Moreover, we suggest that this would enhance the internal motivation of DHH people, potentially giving us three design implications in the early stages: 1) assisting them to perceive each musical element first easily, 2) reinforcing the exploration to catch the cross-modal mapping more intuitively, and 3) keeping them motivated in their exploration with various music-related gamified quests or interaction with others.

Typically, DHH people require sufficient exploration chances to achieve proficiency in the fundamental principles of every musical element; this is an important prerequisite for the accurate conceptualization of cross-modal mapping. Acquainting DHH people with musical elements through visual and tactile representations can aid their perceiving of basic concepts of musical elements, which can be accomplished in several ways. For example, musical elements can be perceived by presenting various sound samples (scales and octaves) and the corresponding explicit visual examples (a ladder expressing height) [1] to make users accept the basic concept of pitch. If DHH people perceive musical elements using these approaches, it will be easier for them to conceptualize how the musical elements correspond to specific visual and tactile elements.

It is also important to design alternative sensory stimuli so that DHH people can intuitively recognize mapping rules when exploring cross-modal mappings. However, Table 1 indicates that there are no definitive rules for the alternative senses that map these musical elements. This suggests that creating an intuitive, cross-modal mapping design is a challenging task. An abstract relation in cross-modal mapping makes it difficult for individuals to intuitively identify the musical elements that are represented by the notes appearing during the playback of a track. Using visual cues, such as color and height, to represent pitches will be helpful, but some participants found it unclear during the initial state of the exploration. Different methods must be developed to explain the notes to DHH people and help them perceive the notes better. Several participants suggested using text for differentiating pitches. We believe these efforts will help DHH people adapt to cross-modal mapping more quickly.

Finally, they must be motivated throughout the exploration process. According to participants, the exploration process can feel like studying music. To prevent them from dropping out, it may be beneficial to make the exploration process enjoyable. For instance, gamification [18, 56] can be effectively integrated during the exploration phase, using elements such as levels and progression [63, 64], as well as achievement badges [20, 23], to enhance musical motivation and engagement. Similarly, peer interaction techniques [10, 22] involving collaborative learning [33, 91] and group problem-solving activities [77, 89] can also be employed, inspiring the process of exploring the musical elements with friends or families. By utilizing these implications, DHH people can complete the exploration process while being motivated, embracing the experience of appreciating music positively without the tediousness of the process.

Letting DHH people explore these things in the system by themselves can make them seek and desire more about the components related to the modalities, leading them to design their own mapping rules and visualizations. This will make them feel closer to music. Previous studies on the personalization needs of DHH people reported automated sound recognition tools [19, 31], hand gesture–based communication devices [92], and vibration feedback [72, 73], which are focused on unimodality. In this study, we proposed a framework for the exploration and customization phases of cross-modalities: sound, visuals, and tactile feedback. Participants customized the cross-modality to enjoy music in a way that felt harmonious to them; thus, they could comprehend and embrace the expression of music more subjectively, rather than solely acquire musical information. DHH people are more free from the dominance of hearing in music appreciation than hearing people, so they can project their own creative expression into music through cross-modality. Therefore, it is essential to evaluate the diverse needs of DHH people and reflect them in the customization.

In addition to accepting the musical expression, participants noted the importance of aesthetic enhancement. Future studies should consider incorporating elements, such as abstract animations synchronized with music [44] or integrating nuanced haptic patterns with game sound effects [99], to cultivate an immersive and pleasurable musical experience for DHH people. Combinations involving the sentiments of music [47, 57], symbolic audio features [51], emotions related to visual features [44], metaphors about tactile features [42], art related to musical features [52], and generating dances from music with machine learning [50] should be considered to enhance the aesthetic aspect of cross-modality. Cross-modal designers and assistive technology developers will need to find more optimized cross-modal matching combinations during the design process, which will provide richer options for DHH people to fully experience music through aesthetic tuning.

Although our framework was designed for the music appreciation experience, it can be extended to enjoy music for various purposes. An increasing number of studies are exploring the cross-modal use of music in teaching and composing music. Emile Jaques-Dalcroze [34] developed a pedagogy that employs the body and voice to enhance musical rhythm, melody, and pitch, and the understanding of music notation and theory, going beyond the boundaries of “sound.” Zoltán Kodály’s methodology [26] underlines the importance of developing an intuitive sense of music by creating one’s music, often in conjunction with dance, to emphasize a holistic artistic experience. The Kodály hand sign, developed by Cuwen [100], is instrumental in ensuring precise pitch recognition when instructing choirs. The music-making and creative process [62] project enables self-directed exploration of music by collaborating and sharing outcomes and by providing personalized learning approaches based on the interests and levels of the student. Although this was not designed for DHH people, research suggests that the teaching and creation of music in a cross-modal environment can considerably enhance the perception and interpretation of music. Note that in our framework, cross-modal mapping rules are not fixed, but rather changeable. Therefore, it can be applied to the cross-modal mapping constructs used in music education and composing; particularly, we expect that customization can be used to motivate music-related activities. Given that experiences and education in diverse settings often leave a profound impression on how music is comprehended and construed [65], our framework may encourage DHH people and underpin their musical pursuits with the system’s scalability in the future.

In this study, we argued the importance of the conceptualization of cross-modal music mapping for the DHH people’s enjoyment of music. However, we acknowledge that some limitations exist. First, although participants were satisfied with the current system, we only used basic musical, visual, and tactile elements in the cross-modal mapping. We also mapped the visual and tactile elements to musical elements one-to-one and designed the basic exploration and customization elements. Our future work will introduce new musical elements, such as harmony [7] or timbre, and more sophisticated cross-modal mapping for DHH individuals. Combining various elements in mapping has the potential to enhance cross-modal representation.

Secondly, this study utilized only a piano to present the melody. This approach was intended to avoid confusion in cross-modal mapping, but it may have limitations in representing other musical elements, such as timbre. Thus, a new approach will be developed that features a seamless cross-modal music conceptualization method, allowing the inclusion of various instruments, such as voice and cello.

Thirdly, this study had limitations in experimenting with the entire spectrum of DHH individuals, each with distinct hearing conditions. DHH individuals with hearing losses ranging from 20 dB to 94 dB were included in our study, but this may not be representative of all types of hearing loss. To enhance the generalizability of our findings, it is necessary to study the effectiveness of the cross-modal conceptualization framework in a more diverse population of subjects with varying hearing abilities.

Lastly, the current version of our system has limitations related to its platform. Natural tool accessibility with portable devices or app-based versions would enable greater immersion in the music-sensory-substitution system, thereby allowing people to enjoy music without limitations. As we continue this research, based on the cross-modal conceptualization verified in this study, we will develop improved visual and tactile feedback devices by referring to various visualization elements with good accessibility, such as the Phazr³ app.

Multimodality. Each musical (melodic and rhythmic) note is visually presented as one dot in the main display panel and tactilely expressed as a vibration. Pitch is visually mapped to the vertical position and the color of dots but is not tactilely mapped. Three methods—frequency, area, and vibration pattern—have previously been employed to express pitch tactilely, but typically several weeks of training are required to be able to adeptly distinguish pitch to sufficiently enjoy a song with these methods. We decided, therefore, to focus on expressing rhythm rather than pitch through tactile stimuli. The range of octaves in our system lies between 3 and 5; thus, C3 (3-octave do) maps to the vertically lowest position, and B5 (5 octave si) is placed at the highest position. Pitch class, such as do (C3, C4, and C5), is represented by the same color—that is, three dos (C3, C4, and C5) appear as red dots. The octaves are distinguished by the height of dots. Loudness can be identified by the size of a dot or the intensity of a vibration. Our system deals with melodic instrument (piano) and rhythmic instrument (drum). As shown in Figure 2, these instruments are visually distinguished by the shape, texture, and color of the dots. The horizontal position on the screen signifies time, while the tactile stimuli do not manifest time. The tempo of the music can be recognized by the speed of the dots being displayed. The time it takes to move from the left to the right edge of the screen is equivalent to four beats when the beats per minute of the music is measured; thus, if the tempo of music is faster, then the movement from left to right on the screen will be faster. The rhythm of music is expressed through the length of notes and rests. In CMP, only the length of the rest is expressed. The length of the rest is visually represented as the horizontal interval between dots, and tactilely as the time interval between vibrations. At this time, the shorter and more frequent the rest, the faster the rhythm.

Exploration. We had users play the drums and piano to represent rhythmic and melodic instruments, respectively. We used one snare drum, and the piano was designed to play 36 notes (12 scales × 3 octaves). As the users play the instruments, they can manipulate the size of the dots or the intensity of the vibrations by adjusting the loudness of the drums and piano. These instruments can be played freely or along with a given sheet music. We wanted users to understand the basic concepts of musical elements, so we presented them with sheet music that included rising melodies, melodies with octave differences, nursery rhymes, and so forth.

Customization. The sensory representation—when musical elements are replaced with visual and tactile senses—can be customized to match users’ preferences. Users’ can realize their preferred visual sensory representation of dots by selecting from among five shapes, four textures, and 11 color sets for piano, and from five shapes, four textures, and a color—which is selected by the user—for drum. These representation options of shape, texture, and color set are included according to the criteria of complexity; a circle, for example, is simpler than a star. Additionally, users can adjust the vertical spacing between pitches and decide whether to use connecting lines between the dots to improve the legibility. The connecting lines between points may clarify the change in pitch more intuitively. With respect to tactile feedback, users can change the sensitivity and intensity. Sensitivity corresponds to how low a sound can be ignored when setting the threshold for haptic feedback. If the sensitivity is set to the highest level, then tactile feedback occurs even for very small sounds. In contrast, if the sensitivity is set to the lowest level, then haptic feedback appears only for loud sounds. Intensity determines how strongly the tool generates the vibrations.

We set the default option of sensory representation that appears before user’s customization to not be too complicated and strong yet still enable the user to feel the change in musical elements. The default dot shape was set to ‘circle’ and the texture was set to ‘none’. The default color set of the piano and the color of the drum were ‘rainbow’ and ‘white,’ respectively. The pitch interval was set at 70% of the maximum, and the connecting lines between the pitches were automatically displayed. The sensitivity and intensity of tactile feedback were set to 30% and 5% of the maximums, respectively.