Telling Time by Vibration

Telling Time by Vibration Sampo Töyssy University of Tampere Department of Computer Sciences Interactive Technology Master's thesis November 2007 University of Tampere Department of Computer Sciences Interactive Technology Sampo Töyssy: Telling Time by Vibration Advisors: Roope Raisamo, Jukka Raisamo Master's thesis, 57 pages November 2007 Touch is an important and often underutilized sense in human-computer interaction. For those with hearing or seeing impairments, touch becomes even more important and may be the only suitable communication channel. Touch can also be used to convey information discreetly. Also time plays a crucial role in our daily lives and therefore telling the time is very important. The two main subjects of this thesis are touch and time. Some applications where time is communicated with a vibrotactile signal have been developed, but the information about them is scarce. This brings up two questions: How to communicate the time with vibrotactile signals and can people understand the signals without training? This is why a method to present time with vibrotactile pulse sequences has been developed in this thesis. Therefore the main subject of this thesis is vibrotactile communication and more specifically: time and vibrotactile communication. Two experiments were conducted which reveal how the method performs. The tests were conducted in order to find out how accurately people can read time from simple sequences of vibration, and how does training affect the recognition rate. The thesis also contains a brief overview of sense of touch, vibrotactility and time. The results reveal that the method is viable and users are able to decode the time correctly with relatively good performance. Keywords and terms: vibration, time, haptic, tactile, mobile phone, vibrotactile, tacton, clock, watch, temporal vibrotactile sequences Acknowledgments I am grateful for the support and advice from my advisors, Roope Raisamo and Jukka Raisamo. Their advice and commitment made the process of creating this thesis easier. My sincere thanks also go to Maila Töyssy and Tero Alatalo, who suggested some improvements. I would also like to thank my significant other, Elli Kotakorpi, for her support. Finally, I extend my thanks to all my friends. Table of Contents 1. Introduction...................................................................................................................5 2. Touch and Computers...................................................................................................8 2.1. Human Physiology and Sense of Touch................................................................8 2.2. Haptic Actuators..................................................................................................14 2.3. Controlling the Actuators....................................................................................16 2.4. Vibration as Information Conveyor.....................................................................18 2.5. Examples of Haptic Communication...................................................................20 2.6. Summary..............................................................................................................22 3. History, Existing Applications, and Related Work.....................................................23 3.1. About the History of Telling Time without Sight...............................................23 3.2. Related Work and Applications...........................................................................25 3.3. Mobile Device Tactility.......................................................................................28 3.4. Summary .............................................................................................................30 4. A Method to Tell Time by Vibration..........................................................................31 4.1. From Numbers to Vibrations...............................................................................31 4.2. The Application...................................................................................................34 5. User Experiments........................................................................................................38 5.1 Goals and Hypotheses..........................................................................................38 5.2 Experiment I.........................................................................................................39 5.3. Experiment II.......................................................................................................40 6. Results.........................................................................................................................43 6.1 Results from Experiment I....................................................................................43 6.2 Results from Experiment II..................................................................................45 7. Discussion...................................................................................................................48 8. Summary.....................................................................................................................51 9. References...................................................................................................................53 1. Introduction Touch is one of the five human senses. When listing the different senses it's usually the last one on the list. It is not surprising, as seeing and hearing are usually more important in our everyday lives. Nevertheless, touch and smell have been irreplaceable to our survival since we've had to distinguish the edible substances from harmful ones and touch enables us to use tools with great dexterity. Therefore new applications for touch should be studied and developed. Time is important to us and some say that it has become too important. Nevertheless, the need to tell time concerns everyone. Most of us can tell the time by checking our wrist watches or mobile phones. Sometimes it would be nice to tell the time without looking anything at all. Some people have impairments and they cannot see the time so they have to use alternate methods. There are tactile clocks, clocks which speak the time, clocks which can beep the time, and finally – clocks which can vibrate the time. However, vibrating clocks are rare. When the writing of this thesis began, I was only able to find one actual implementation which was the VibeWatch [2002]. The scarce information about vibrotactile time was an inspiration to this thesis. The basic idea of this thesis is to connect vibrotactility and time. By combining vibration and time, we can communicate time in a simple way via touch. That is why a method to present time by vibration was designed and tested. Two experiments were conducted to evaluate the method. There are two questions to be answered: 1. Can people understand time from vibration without training? 2. How does training affect the results? There are at least three different possible scenarios where vibration encoded time could be used. The most obvious user group is the deaf-blind. They have no way of hearing or seeing the time, so the information has to be conveyed in another way. The second group is the blind who need to tell the time without voice synthesis or other auditory cues. The third group is the people with normal hearing and vision who need to tell the time discreetly. Anyone may be temporarily unable to use audio or see the time due to situational impairment which may be caused by ambient noise or lightning conditions. 5 The application and the method were developed and tested on a mobile phone since it provides a fairly cheap platform from both developer and user perspective. The application is straightforward to reproduce on any programmable mobile device equipped with vibration. The first experiment was conducted as a proof of concept to see if the method is viable. The users were given very little training in the first experiment to determine the recognition rate for beginners. The second experiment was larger and studied also the learning curve. In the second experiment the test sessions were longer. The ultimate goal was to achieve 90% overall recognition rate. This threshold was chosen since a performance requirement around 90% is quite common in similar tests, for example, Tan [1997] used a performance requirement of 95% in several tests in her haptic Morse experiments. A couple of definitions are given below. According to Lederman [1997] the term “haptic” also includes sensory input from joints, muscles, and tendons (also called kinesthetic information) in addition to cutaneous information. The stimulus applied to the skin is received by the receptors in the skin and then turned to cutaneous information. This thesis favors the terms “tactile” or “vibrotactile” since, e.g., a vibrating mobile phone produces very little kinesthetic information if any at all. Also the concepts “active” and “passive” should be defined. Active means that the device is able to produce interactive feedback such as vibration and that the feedback can usually be controlled by software. Passive means that the feedback is noninteractive and depends on user activity. A vibrating mobile phone would thus be an active and a static keypad would be a passive device. The second chapter about the basics of haptics has been divided in six sections which travel through the field of human-computer interaction which has two sides: humans and computers. The basics of human sense of touch are described first, and after that the actuators through which the computers can communicate. Basics of actuator control and communication are also discussed. Finally, a selection of different haptic communication methods is described and the chapter ends with a summary. Time is discussed in historical perspective at the beginning of the third chapter. Related work is also discussed. At the end of the chapter, mobile device tactility is discussed and the chapter is closed with a summary section. 6 In the fourth chapter the actual method of telling time by vibration is described along with the application used in the tests. In the fifth chapter the two experiments are described and in the sixth chapter the results are revealed. The seventh chapter contains discussion about the method and the results. The eighth and final chapter is the summary of this whole thesis. 7 2. Touch and Computers This chapter covers the basic background of human touch. Methods of conveying information via that channel are also discussed. The chapter first discusses touch as a human sense from a more theoretical point of view. It then approaches the topic from the perspective of haptic actuators and how the computer is able to control them. After that these two perspectives are combined into a discussion about conveying information via touch. The chapter ends in the section where some different vibrotactile communication methods are presented. Most of the communication methods are more complex than the method of presenting time by vibration developed and tested in this thesis. However, they show how diverse the field of vibrotactile communication really is. As this whole thesis is about conveying information via touch it is important to know the basics behind this communication channel. The chapter, as this whole thesis, centers around vibrotactile feedback, but also other haptic modalities are discussed when deemed appropriate. 2.1. Human Physiology and Sense of Touch Humans have five major senses: vision, hearing, smell, taste and touch. Touch is the sensation produced by mechanical, thermal, chemical or electrical stimulation of the skin and body. The sense of touch is important to us humans as we depend on our skillful use of tools. This is why this chapter begins with a physiological description of touch. The hairless part of human hand has about 17 000 mechanoreceptive tactile units. There are four different types of tactile afferent units when classified by adaptation speeds: fast-adapting I (FA I), fast-adapting II (FA II), slow-adapting I (SA I), and slow-adapting II (SA II). Fast adapting (FA) means that the receptors react better to transient stimulus such as hitting a palm with a ball point pen. Slow adapting (SA) means that the receptors are more sensitive to prolonged stimulus and they can sense immobile objects touching the skin. There are also four kinds of nerve endings related to the tactile units: Meissner corpuscles (FA I), Merkel cell neurite complexes (SA I), 8 Pacinian corpuscles (FA II), and Ruffini endings (SA II). These receptors are used to receive tactile sensory information (cutaneous information). The details are summarized in Table 1. The structure of the skin and the different receptors can be seen in Figure 1. [Lederman, 1997] Name Ending FA I FA II SA I SA II Cold Hot Nociceptor Meissner Pacinian Merkel Ruffini - - - Best response 40 Hz 200 - 300 Static Static Hz pressure / 0 pressure / 0 – (Verry / (flutter) / Mazonne) 30 – 40 (buzz) / – 10 Hz 10 Hz Hz 130 – 150 Hz Decrease in Increase in temperature temperature Intense stimulation (pain) 20 – 100 Hz 100 – 1000 Hz 0 – 10 Hz 0 – 10 Hz Mean receptive field area (mm2) 12.6 101.0 11.0 59.0 2-9 2-9 - Relative Depth Shallow Deep Shallow Deep - - - Number of specialized endings per tactile unit 12-17 1 4-7 1 Proportion 40% 13% 25% 19% Frequency range (Mazonne) Table 1: Human Cutaneous Receptors, adapted from Lederman [1997], Mazzone et al. [2003], and Verry [1998]. 9 Figure 1: Tactile units and their effective areas in the human skin. In an interview by René Verry [1998] Lederman also mentions three other receptors. The first two are the temperature receptors for both cold and hot. The cold receptors sense decrease in temperature and hot receptors sense increase in temperature. The temperature receptors have no specialized nerve endings and they have their own response ranges which overlap slightly. Their receptive fields are approximately 1 to 2 mm in diameter and they have a density of 1 to 5 cold-sensitive receptors per cm 2 and 0.4 warmth-sensitive receptors per cm2 [Mazonne et al., 2003]. The third receptor is the nociceptor which responds to high intensity stimulation, also known as pain. Mazonne et al. [2003] describe three different types of pain receptors. Mechanical nociceptors detect sharp mechanical stimulation such as sharp objects. Thermal nociceptors respond to extreme heat or cold. The third type is a polymodal receptor which responds to several different kinds of noxious stimuli. However, pain is a more complex sensation than the information delivered by a single receptor. 10 After the stimulus has been observed by the nerve endings, the signal must travel to the brain to be processed. There are five types of neural fibers (axons). A-alpha is a fast conducting fiber which is associated with the kinesthetic sense. A-beta is a moderately fast fiber that is used to transmit both kinesthetic and cutaneous information from mechanoreceptors. A-delta fibers are also moderately fast and they deliver signals from temperature and sharp pain signals from pain receptors. They also transmit crude touch information. C-fibers are slow and they are responsible for temperature signals and burning pain. The fifth type is A-gamma and it transmits information from muscle spindles. [Shaffer & Allison, 2007] The signal from a receptor can travel actually through two different pathways to the brain. The first one, the Dorsal Column-Medial Lemniscal (DCML) pathway, is usually used by touch, vibration, and proprioceptive information from the limbs. The spinothalamic pathway mostly carries signals from temperature and pain receptors. The DCML pathway is more efficient and accurate and is thus usually used to deliver the more fine grained information from the delicate touch receptors. Usually there are more neurons on the spinothalamic pathway and the fibers on the DCML pathway are larger. However, information from touch receptors also travels in the spinothalamic pathway but it may be mixed with signals from temperature or pain receptors. The role of the spinothalamic pathway in the sense of touch is still largely undetermined. [Iggo, 1987] The touch sensation is also centrally controlled. Stimulus on the skin does not necessarily cause excitation in the somatosensory cortex. This effect is produced by excitatory and inhibitory synapses and it can completely or partially block incoming signals [Iggo, 1987]. This can also be used to raise the contrast between the stimulated area and other regions. This means for example, that a sweater which causes uncomfortable itching at first, will be comfortable after a while as the brain turns down the stimulus associated with the sweater. Among other things, this mechanism enables the brain to concentrate on the usually meaningful and important stimuli instead of the ambient noise. Similar mechanisms are also associated with other senses. Sensitivity of human skin has been determined with various methods. One of the methods is to press calibrated nylon mono filaments on the skin so that they barely bend. Different body parts have different sensitivities. The most sensitive parts are the face, torso, fingers, and lower extremities. Females and males have different thresholds as do the different sides of the body. Because of all these factors, the threshold varies 11 greatly between 5 to 355 G. Cutaneous sensitivity has also been described in Weber fractions. The studies have obtained following values: 0.14 for static pressure, 0.2 for impulse stimulus, and 0.2 for 160hz vibratory bursts. Weber fraction represents the Just Noticeable Difference (JND). When weighting a stone, a Weber fraction of 0.1 means that the the weight needs to change at least 10% for the observer to notice the difference. [Lederman, 1997] Like many other senses, touch might also be affected by aging. For example Brammer et al. [1993] found out that the threshold mediated by the Pacinian receptors in the fingertips increases 2.6 dB per 10 years. Some studies have revealed how Pacinian and Meissner's corpuscles change when a person ages. For example the number of Pacinian corpuscles seems to decrease along with their sensitivity. The concentration, size, and number of Meissner's corpuscles also seem to go down. Also, the detection threshold seems to get higher. [Shaffer & Allison, 2007] Spatial sensitivity has been measured with the “two-point touch” method where the observer has to answer whether one or two points are touching his skin. The resulting measurement is the minimum distance where the observer can sense the two points instead of one. This distance can be as low as 2-3 mm on the fingertip but almost 45 mm on the calf. The spatial sensitivity clearly depends on the body locus. Another method is the “point localization”. In point localization, the skin is first touched with a pointed object. Then the skin is touched again. The second touch may be a certain distance away from the first. After this the user has to answer whether the two touches landed on the same location or not. The closer the two points are to each other, the more likely they are experienced as the same point. This threshold is also dependent on the body locus. Localization is the most accurate on fingertips and the least accurate on the back. [Lederman, 1997] The sensitivity to vibration depends on the location and frequency. Van Erp [2002] defines a general range from 20 to 500 Hz. Lederman [1997] defines the range from around 40 to 300 Hz. Verry [1998] mentions 40 Hz as the lowest threshold for best response. So there are different views about the sensitivity, which is not surprising given the complex nature of human senses. The lowest detection threshold can be found on a glaborous skin (non-hairy skin) and when the frequency is around 200 to 250 Hz. Fixed elements around the vibrating element and extended duration of the stimulus also 12 lower the threshold. Weber fractions for resolving vibrotactile frequency vary between 0.20 and 0.25. Also it has been determined that two 1 ms pulses need to be at least 5.5 ms apart to be perceived as successive. [van Erp, 2002] Another view to spatial recognition is the single-unit representation of the stimulus. The units are the different mechanoreceptor populations in human skin (see Figure 1). In some studies the signals from these mechanoreceptors have been recorded. In one study braille-like patterns were moved on the fingertip skin of a monkey [Lederman, 1997]. The fidelity of different receptors is linked to their receptive field area. In these studies slow adapting receptors provided the most detailed perception followed by the fast adapting receptors. Pacinian receptors were the most inaccurate. Only three channels have been identified in a monkey but they are similar to human receptors. It is interesting how a blurred vision is comparable to touch. A section of skin has a limited resolution since there are only so many spatially resolvable points whereas the retina is vastly more accurate. When the effective spatial resolution of vision was matched to that of a fingertip the recognition results were very similar. For example braille was determined to be more legible to blurred vision and normal touch than embossed Roman alphabet. [Loomis, 1981] It seems that visual stimuli tends to dominate other stimuli even if they had greater intensity. Visual signals seem to override auditory, tactile, and proprioceptive stimulus [Cooper, 1998]. It is argued that the phenomena rises from a mechanism designed to compensate for the low alerting capability of visual signals. Therefore, if two signals are used to communicate simultaneously with the user, the reaction time will be determined by the visual signal even though reaction to auditory signals is faster by itself. 13 2.2. Haptic Actuators Haptic actuators are the link between human senses and interactive feedback. Therefore their attributes define the basic nature of the interaction. If an actuator is ill suited for the task, the quality and usefulness of the application will deteriorate. The actuators have been divided according to the basic mechanism which produces the physical displacement which is then sensed by the receptors in the skin. Most of the actuators presented can be used to produce vibrotactile stimulation, but some are better suited for the task than others. One of the most common actuators providing vibrotactile stimulation is a simple electric motor with a rotating eccentric mass producing vibration. The device is also called a rotary mass vibrator (RMV). The RMVs come in different sizes and they have different attributes. Their rotational speed can be adjusted and some also can be spun backwards. The RMV is practically ubiquitous in mobile phones. This is because it is cheap to produce, easy to control, and it does not require a lot of power. Before the vibrating mobile phones the haptic channel was not used much as there were few every day devices which would produce active haptic stimuli. However, passive haptic cues, such as small embossed dots in numeric keypads, have been used for a long time. Voice coils [Voice Coil, 2007] have also been used to produce vibrotactile feedback. There are linear and rotary voice coil actuators. Unlike the rotating electric motor in an RMV, linear voice coil motors produce a stroking motion similar to piston movement in a car engine. Since voice coils can be controlled with great accuracy they are also used to position the arm in a modern hard disk. Motion from a rotary voice coil motor is similar to a conventional RMV actuator. Voice coils can also achieve high acceleration. Piezoelectric devices can produce vibration or move pins in a linear fashion too [Piezo, 2007]. They are based on crystals which can expand when they are electrified. They can produce relatively strong forces and their bandwidth is high. Piezoelectric actuators are used in braille displays. Poupyrev et al. [2002] claim to have developed a piezoelectric actuator called “TouchEngine”, which has several advantages compared to conventional motors. The size of the actuator is small and due to fine layered design, it does not require a lot of power to produce motion. On the other hand, the displacement produced by the actuator is small but it can be controlled very precisely. 14 Electrostatic actuators have two subcategories: motors and electrostatic “displays”. Electrostatic devices are based on the attraction and repulsion of different charges. Small electrostatic motors can be easier to produce than small conventional electric motors. This is why electrostatic actuators have been used in micro scale applications [Sidobre & Hayward, 2003]. The electrostatic actuators need to be relatively small since the electrostatic effect decreases with the square of the distance between the two charged bodies. Yamamoto et al. [2004] developed an electrostatic tactile display which consists of a thin conductive film slider with stator electrodes which excite electrostatic forces. Basically the user moves the slider with a finger and experiences sensations which are interpreted as tactile textures. In this application a sensor is used to read the texture of a material and the resulting data is then fed into a processing unit which produces the control signals for the electrostatic display. They reported a successful discrimination rate of 79% in a test where the user had to discriminate between different textures. A more exotic approach is to use Shape Memory Alloys (SMA) to produce motion [SMA, 2007]. SMA-wire can change shape when the temperature of the wire changes. The displacement is relatively small so mechanical systems are used to amplify the stroke. In comparison with electronic motors, the bandwidth is usually low and forces are small. However, the devices can be simple as there are few moving parts. Even more exotic actuators have been developed, but their practical usefulness outside research is limited. Electroactive polymers can change their shape according to the supplied voltage. They can undergo large amounts of deformation while sustaining a formidable force. Electroheological substances can change their viscosity when the supplied current is varied. This means that their rigidity changes when the electric field in which they are immersed in is changed. Magnetorheological substances change viscosity when the magnetic field affecting them changes. As a side note, some cars use magnetorheological fluids in their shock absorbers to provide dynamic shock absorption. One such car is Ferrari 599 GTB [Classic Driver Magazine, 2007]. 15 Pneumatic systems can produce extremely strong forces but they are cumbersome and unpractical compared to electric systems. They require a pressurized air supply which usually means a compressor or a pressurized storage. Rizzo and Messeri [2004] developed a glove with inflatable air bladders. They claim that using air bladders instead of miniature pneumatic cylinders is more simple and more economical. The glove is used to simulate grasping sensations. They demonstrated a statistically significant change in grasping success in the presence of haptic feedback. Air puffs can also be used to stimulate the skin. This method is actually used in some theme parks. For example, the author of this thesis has personally experienced the spooky sensation from a puff of air in the theme park called Särkänniemi in Tampere, Finland. The driving force behind the air puffs is supposedly some kind of pressurized air system with an electronic valve. Suzuki et al. [2002] have produced a different system which uses jets of air and a small racket for the user. When the jet of air hits the racket, the user perceives a simulated force. The advantage of the air based systems is their untethered nature. Plain electricity has also been used to stimulate, but it is rather invasive and most people would probably consider small electric shocks as annoying or even painful. However, one group used electric shocks in their game called Painstation [2007]. In this particular game the effect from the non-lethal electric shock is designed to be discomforting. It is not surprising that the electro-magnetic actuators are among the most popular. This is due to their performance, price, and easy handling. These include electric motors, piezoelectric devices, and electrostatic actuators. Conventional motors can produce vibration, move pins in linear fashion, or brush a rod against a finger to produce sense of friction or slip. There are many types of actuators but only a few of them are used in consumer electronics. This is because only a few of them meet the commercial requirements at the moment. Actuators used in commercial products have to be cheap, reliable, portable, safe, and easy to integrate. 16 2.3. Controlling the Actuators The actuators connected to a computer system need to be controlled by software. Some requirements arise from the nature of the application. The computer needs to be powerful enough for the required task, the lag should be low enough for the required application, and the overall system should have the necessary real time capabilities. Simple applications, such as the vibrotactile time described in this thesis, do not require a lot of computing power. The real time requirements are also low. Therefore many simple applications using tactile and even kinesthetics feedback are able to run even on low powered mobile devices. However, the mobile devices are changing fast and the processing power of a current desktop computer could be available in a standard mobile phone in a couple of years. On the other hand complex virtual environments with haptic feedback require a lot of processing power since the devices are often complex and they have as much as six degrees of freedom in input and output. Also the update frequency of the haptic rendering pipeline should be at least 1000 Hz, since users can distinguish the different rendering rates between 500 Hz to 1000 Hz [Otaduy & Lin, 2005]. The most known example is the SensAble Phantom [2007] device which is controlled by a computer. The Phantom device is usually used to convey realistic touch feedback from a virtual environment. To convey a realistic illusion of objects, materials, and their interactions, the computer must process both the visual environment and the touch interactions fast enough. The recent advent of affordable multi core processing will probably make even more complicated applications available to a much larger population of researchers. The inexpensive processing power can be coupled with affordable devices such as the Novint Falcon [2007] to produce complicated and feature rich applications. The main topic of this thesis, vibrotactile communication, does not usually require a lot of computing power since there are no real time simulations or other complex tasks. Mapping different symbols to vibrotactile signals requires only a few operations. Also the necessary actuator only receives commands from the hardware and does not produce information about the orientation of the device. 17 2.4. Vibration as Information Conveyor The process of receiving and decoding streaming information from a vibrotactile device is complicated and has several phases. First, independent signals must be recognized from the stream of information and they have to be interpreted as meaningful pieces of information. After that, the pieces of information must be stored in the short term memory for later use and at the same time new signals must be read [Tan et. al., 1997b]. Finally, the small pieces may have to be assembled into a larger whole. So there are two factors which must be considered: how do humans process and understand sequences of vibrotactile pulses and what is the capacity of the short term memory to handle information from a vibrotactile source without training. These factors are different from images or text where the target may be re-examined all the time. The software which produces the pulse sequences may be simple, but the human task to decode those sequences is not that straightforward. The first factor in reading information from a temporal vibrotactile pulse sequence is the ability to separate signals temporally. It has been determined that there has to be a gap roughly 10 to 50 ms long for two signals to be perceived as separate [Tan et. al., 1997b]. Also van Erp [2002] claims that gaps and signals need to be at least 10 ms long to be perceived. The setup time for the stimulus also needs to be taken into account since many actuators, such as rotary motors, have a spin up time. If the amount of information is low, this should not be a problem, since longer gaps will not make the sequence overly long. Also longer gaps will probably give the user more processing time for memory tasks. It may be even so that the bottlenecks would occur due to the relatively slow speed of handling temporally encoded tactile stimulus rather than in difficulties separating a gap between pulses. Additionally, very short pulses and gaps may require special equipment with fast response times and dependable real-time capabilities. Longer gaps will probably lower the effects of temporal enhancement [van Erp, 2002]. Van Erp [2002] states that a maximum of four different magnitudes can be used in vibrotactile signals for them to be discriminable. When controlling the RMV in a simple mobile phone, the only attribute is usually the length of the vibration. However support for additional attributes is emerging for some platforms like S60 3rd edition [CHWRMVibra, 2007]. Amplitude and intensity may also come into play as uncontrolled variables, since the motor producing the vibration has a spin up time. 18 When the time is shorter the sensation is also milder since the motor does not necessarily have time to reach full speed. Van Erp's rule for a maximum of four different signals is likely to be true also in the case of RMVs. Van Erp [2002] classifies vibrotactile data representation to four different channels: amplitude, frequency, duration, and location. Amplitude is the intensity of the signal as it describes the magnitude of oscillation in a wave. Furthermore, van Erp claims that amplitude and frequency offer limited possibilities in encoding information since they have a limited amount of different discriminable levels. He tends to favor temporal and location encoding. Temporal encoding should be efficient when designing a single element vibrotactile display. The application described later in this thesis uses temporal patterns from a single element to convey the information. Van Erp [2002] notes that the guidelines he has presented may not be viable in applied environment since the data behind them has been acquired in optimal laboratory conditions. However, it can be presumed that if they are used with caution and with considerable tolerances they can be used in first design stages as guidelines. The famous argument by Miller [1956] is that humans have a forward memory span of seven items plus or minus two. Tarnow [2005] has presented data from experiments conducted by Rubin, Hinton and Wentzel. In their experiments they asked students to learn a list of words or word-pairs. After they had learned the list they were asked to repeat it after a certain time has passed. Right after they had learned the list the recall probability was 0.944. After six seconds it was 0.646 and after twelve seconds 0.434. In the experiments described in this thesis, the time between the first number and the moment to answer can be over 13 seconds. The data required in the task is only a pair of two-digit numbers so it should not be a major problem to remember them. Geldard and Sherrick [1965] developed a system called the Optohapt. The Optohapt had 10 vibrators all over the subject body, which could be used in different ways to represent different symbols. Unlike the Vibratese system by Geldard [1957], the Optohapt could activate several vibrators simultaneously. The system was also used to present letters and words. It was observed that people have difficulties remembering 19 previous letters when they were presented with a vibrotactile array covering the whole body. This might be due to the poor short-term memory for touch patterns and their diffusion in time or due to the diffusion of the stimuli over the whole body, or even both. The original role of touch is to experience our environment and gather information from it. It has not been necessary to use touch as a channel for language or for any other abstract symbol system in the past since vision and hearing usually receive these signals. However, human brain adapts well to different situations and requirements. At the moment we are trying to use new technologies to make us humans more efficient and more powerful. It is not clear how touch should be used in this task. It may be that touch based communication of formal languages or information constructs can not be made sufficiently efficient, but sometimes it may be the only option. 2.5. Examples of Haptic Communication In the fifties a vibratory communication language called Vibratese was developed at the University of Virginia [Geldard, 1957]. The system consisted of five actuators situated in five different body locis. The actuators had three intensities and three durations thus resulting in 45 different signals. The vibration frequency was fixed at 60 Hz. The mapping of symbols was carefully designed so that the most common symbols would be the most simple ones to decode. They also left five elements unused: those with the longest duration and medium intensity which dropped the total combinations to 40. It took about 12 hours for the subjects to learn the single letter representation. The highest rate achieved was 38 words per minute. An average person can read around 200 words per minute [Gould et al., 1987]. Use of vibrotactile Morse code as an alternative communication method for the deaf-blind was suggested at least by Zuckerman [1984]. When the user presses a key on the keyboard the system repeats the key press in Morse code. The Morse code had to be enhanced a bit to provide the additional characters necessary with computers. A line of text could be read from the screen with Morse code by pressing a key, just like a sighted person would steer his eyes on the line of text. The user could read the screen by triggering any of the 25 lines of text with corresponding keys running from A to Y. The 20 method was praised by Zuckerman in the year 1984: “Provision of a screen reading mode that allows access to the same characters that may be perceived visually by a sighted person gives the deaf-blind user unlimited capacity utilizing the computer”. Today the situation is more complicated because of graphical user interfaces. The sighted have a much more richer experience since computers are able to produce beautiful graphical user interfaces. This means that there are some new problems to be solved. Tan and others [1997] have studied how people learn to receive Morse code via different methods. While Morse is usually received aurally, hearing impaired are able to put their hand on the speaker to receive Morse tactually without any special devices. The participants in Tan's tests needed to recognize letters, random three letter sequences, common words, and finally sentences. The test subjects were either experienced Morse operators or beginners who received some training. He used three different modalities in the tests: finger motion, vibration, and audio. The motion actuator was a custom built device which moved the finger with a magnetic servo. The vibrotactile unit was a electrodynamic minishaker with a vibration set to about 200 Hz. The vibration pulses were gated with a square wave signal. The auditory condition used normal headphones and the signal was the same 200 Hz square wave signal which was used with the minishaker in vibrotactile condition. The results showed that audio was better than vibrotactile and vibrotactile was better than finger motion. Also the experienced subjects fared a lot better in the tests and they could do tasks which the beginners could not. Their experience with auditory Morse code was transferred very effectively to other modalities like finger motion and vibration. There are people who can understand language using the Tadoma method where the receiver touches the face of the person who speaks [Sherrick, 1991]. The listener interprets the language from numerous different patterns the face of the speaker produces. This example is not strictly vibrotactile since the receiver gathers information from several channels: vibrations from the skull, movement of facial muscles and bones and even puffs of air. Artificial Tadoma emulation has also been developed, but it has not been capable to deliver the tactual stimulation at a rate comparable in human to human communications [Tan et. al., 1997b]. 21 Optacon is a device which translates images to vibration patterns. It provides access to all written material in vibrotactile form. The two basic elements of the device are the camera and the 24 by 6 array of vibrators which form a vibrotactile display of 144 units. The camera images a letter and the array vibrates the corresponding pins. The users can read at a speed of 30 words per minute. [Lederman, 1997] 2.6. Summary The different areas of vibrotactile communication have been covered here starting from the computer hardware and ending as a perceived sensation in the human brain. Depending on the setup, the methods, elements and the pathways vary. However there are always five elements: the nervous system, the receptor, the actuator, the hardware controlling the actuator, and the software controlling the hardware. These elements form the basic structure of computer to human vibrotactile communication. Different applications of vibrotactile communication have also been discussed. At the moment most of these applications are research projects. This was motivating for this project, where a meaningful everyday vibrotactile application is constructed without any specialized equipment. However, it also indicates that it is not easy to use touch to convey information or otherwise there would already be a lot more commercial applications in every day use. 22 3. History, Existing Applications, and Related Work In this chapter we move closer to real applications and experiments related to touch and computers. The first section is about history of non-visual time telling. It highlights some methods used in the past and some of them are still used today. In the second section some different applications on the field of haptics are described. Some of the applications are closely related to the application described in this thesis and others are just interesting by themselves. They include the VibeWatch [2007] which communicates time vibrotactily close to the method and application described in this thesis. There are also more conventional approaches such as a wrist watch which can be touched to feel the time. The third section in this chapter is about mobile devices and tactility where the current state of mobile device tactility and the ideal mobile actuators are discussed. Finally, the chapter is summarized in section 4. 3.1. About the History of Telling Time without Sight Before the advent of affordable personal chronometers there were several different ways to announce the time. Actually wrist watch came in to use just after the First World War, so personal time telling is a relatively new invention. Telling time from other cues than a clock face or from a number display has a long history. The progress can be compared to that of computers: from centralized time to personal time. Interestingly, today the time is in a way centralized again: most computer systems can be set to query the exact time from Network Time Protocol Servers (NTP-servers) [Mills, 2003]. One of the old methods includes a bell - a simple percussion instrument found in many places. In ships the time was announced by ringing a bell with a certain code [Ship's Bell, 2007]. The bell was rung every 30 minutes which was determined by the watchman with a 30-minute hourglass. Once all the sand had flowed down, the watchman flipped the hourglass and rung the bell. The bell was rung once for every 30 minutes passed in the current watch and the watches were four hours long. For example, if a watch started at 12:00 in the afternoon and the bell was rung three times, the time was then 13:30. There are a total of eight watches in a 24-hour day: midnight watch 23 (00:00 – 04:00), morning watch (04:00 – 08:00), forenoon watch (08:00 – 12:00), afternoon watch (12:00 – 16:00), first dog watch (16:00 – 18:00), second dog watch (18:00 – 20:00), and first watch (20:00 – 00:00). Even today ringing of the bell regulates the daily routines on many ships. Time has been announced with bells in cities too. One of the most famous clocks in the world is the Big Ben of London. You can tell the time without seeing the clock face as the number of sequential bell rings on the hour represents the number of the hour. Big Ben also rings a different sounding bell on each quarter hour so people can tell if it is 15, 30 or 45 minutes past. [Big Ben, 1997] Telling the exact time at sea was difficult before the invention of accurate chronometers in the 20th century. To make matters worse, it was very important that ships had accurate time for navigation purposes. Also, pendulum clocks were not usable on rocking ships. Because of this, the time was sometimes signaled to ships from ashore. One of the oldest signaling devices is the “Noon Gun” in Cape Town, South Africa. It began firing the time in 1806. Interestingly the time is usually read visually by observing the smoke from the cannon as light travels a lot faster than sound. The delay between sight and sound from a distance of 3000 meters is practically almost nine seconds (299792458 m/s vs. 344 m/s). [Noon Gun, 2005] There are also mechanical wrist watches called “minute-repeaters” or “striking watches”, which can also signal time with audible signals. The repeating function was usually powered by cocking a spring loaded system which then repeated the time by the minute. For example, there could be two different tones, A and B, which could be combined to create 3 different signals. Tone A signals amount of hours and tone B the amount of quarters. When B was combined with A it would signal the amount of quarter hours resulting in signal C. Thus 4:16 would be played as: A-A-A-A-C-B. The repeaters were developed before electricity, when illuminating a dark environment required some work. The repeater does not usually differentiate between AM and PM and they are designed for the 12-hour clock. Theoretically repeaters could also have vibrated the time if given enough power by the cocking assembly. However, no applications using such mechanism were found. Also, it may be possible that the small hammers and gongs ringing inside the watch could have been felt. [Perez, 2001] 24 Without a minute repeater you would have had to light a candle or some other external light source to read the time visually. Nowadays clocks have fluorescent arms, little LEDs to provide light, and LCD displays are usually back-lit. Also, switching on an electric light source is a lot more simple than lighting a candle. An example of a modern wrist watch with a minute repeater is Citizen BL9009-54F [Citizen, 2007]. 3.2. Related Work and Applications Several methods have been used to convey the time with vibration. Vibe Watch [Vibe Watch, 2006 and 2002] uses short and long pulses to encode the time. It has been developed by the Royal National Institute of Blind People [RNIB, 2007]. The user can activate the unit by pressing a tactile button on the front. There are also options to set the time with vibrotactile feedback. Unlike the method described in this paper, Vibe Watch uses 12-hour clock. 24-hour clock has a wider range of numbers but it does not require AM/PM information. The 12-hour clock is dominantly used only in the United Kingdom, Australia, Canada, New Zealand, the US and the Philippines. The structure of the Vibe Watch time sequence is as follows: 1 long pulse for AM and 2 long pulses for PM, 1 to 12 short pulses for hours, 0 to 5 long pulses for tens of minutes, and 0 to 9 short pulses for minutes. The AM/PM part and hour can not be zero (no pulses) but the minutes after them can be, and if they are, no pulses are played. For example: 2 long pulses, 4 short pulses, pause, 5 short pulses represents a time of 4:05 PM. Vibe Watch is a small separate device which costs around 50€. The research done for this paper did not reveal any information about the user base or viability of the Vibe Watch. The organization responsible for the device was contacted, but they only delivered the operating instructions. One of the main goals of this paper is to answer the question: can users really tell the time from vibration pulse sequences. Positive results would also support the viability of the Vibe Watch. Interestingly, a J2ME program, which is strongly related to the application presented in this thesis, was found on the Internet during the last weeks of writing. Had it been found sooner, the two methods could have been compared as the application presents time a bit differently. The creator of this application is Che-Wei Wang. The Wang Haptic Clock [2007] (named after the creator) uses a 12-hour clock. It first uses 1 to 12 vibrations to signify the amount of hours. After that it signals the amount of 25 minutes by dividing the current minutes by five and then plays the resulting amount of vibrations. So for example 8:36 would be 8+7 vibrations. The method is simple but the accuracy is limited to 5 minutes. Also the application vibrates minutes quite fast so it is easy to lose track. However, that should be easy to reconfigure. Tissot has designed a watch which has vibration combined with spatial sensing on the clock bezel [Tissot, 2007]. Time can be read silently by moving a finger clockwise on the clock bezel. The hours are marked with small knobs like on a normal clock face. When a finger travels on the bezel, a long vibration means that the finger is on the hour and short pulsating vibration indicates that the finger is on the minutes. The watch has a clock bezel which senses the finger position and a small RMV inside. When the finger is sensed on the position where a certain clock arm is, the RMV is started accordingly. The alarm on the clock can also be set by touching the clock bezel. Tissot watches are known to be expensive and this one with the additional vibrotactile technology is around 400 euros in 2007. Citizen BL9009-54F is a modern wrist watch with a minuter repeater. The encoding of the time is a bit more complicated than in Vibe Watch. First, a high pitch sound signals the number of hours (signal A). After the high pitched hour signals the amount of quarter hours is signaled with a signal that combines high and low pitch (signal B). Finally the additional single minutes are signaled with a a low pitch (signal C). When the time is 4:48 the watch would produce the following sequence: A-A-A-A-B-B-B-CC-C. The watch uses a 12-hour clock and does not distinguish between AM and PM so the user has to check whether it is AM or PM. When moving towards mobile devices and vibrotactility, a vibrotactile TFTdisplay from Immersion Technologies can be mentioned. The device combines a modern LCD and haptic interaction [SenseTouch, 2007]. The screen can produce tactile stimuli when the user presses a button on the screen or drags a slider. The SenseTouch technology is claimed to improve usability in noisy or distracting environments. The technology behind the screen is quite simple. One or more actuators and a control board are installed inside the display chassis. The controller connects to the host via a USB or serial connection. When the user touches the screen, the host computer receives the event, a command can be sent to the actuator controller and the actuators can be fired with the desired parameters. Screen sizes can vary from 2 to 19 inches, so in theory the technology could be used from mobile phones to information kiosks. The vibrotactile 26 representation could be a lot more versatile since location could be used in addition to temporal patterns. Still, mobile phone use might be difficult as phones have very limited capacity and actuators would add to weight and enlarge dimensions. It would be surprising if the average user would favor better vibrotactile feedback over small size. However touch enabled screens might become more common (as of writing in 2007 the iPhone from Apple is one of those products) and the existing simple RMVs could be used more effectively. Also, a touch sensitive vibrotactile display could be used to produce a virtual tactile clock. A more novel approach is the vibrating shoe sole. Fu and Li [2005] implemented a system where three RMVs were installed in a shoe. The RMVs are used to alert the user when certain changes happen on the stock market. The vibrating shoes have a wireless connection to a host computer which can activate the vibrators when it detects preset movements on the stock market. The description of their implementation is limited but the motivation is clear: foot soles are sensitive enough to receive stimuli from a couple of separate actuators. Just how sensitive are they? Would it be possible to transmit braille via 6 RMVs (3 on both soles)? Time could be quite easy to present with vibrating shoe soles. Others have also found foot soles as potential information receivers too. Frey [2007] has developed CabBoots - shoes that guide the user haptically. They rely on the phenomenon in which the user tends to correct his path when he encounters the slope on the edge of the path. CabBoots replicate this virtually. When the user is close to the edge of the virtual path, servos push the sole so that the user gets the illusion of the path edge. Needles to say, this application is quite complicated with servos and different sensors. Ideo [2002] has developed a similar system which relies on simple vibrators on toe rings to tell the user where to turn. Brown and Kaaresoja [2006] conducted an experiment where the user had to recognize incoming communication with only vibration. The attributes were the type of alert (voice call, text message, or multimedia message) and priority (low, medium, or high). The type was encoded in a rhythm. For example, seven short pulses indicated a voice call. The priority was encoded by roughness or by intensity. This resulted in total 27 of 18 possible tactons. Overall recognition rate for the rhythm+roughness condition was 52% and 72% for the rythm+intensity. The recognition rate for roughness is only mediocre, but the intensity worked better. The different rhythms were recognized with 95% accuracy. Another study by Kaaresoja & Linjama [2005] was aimed to find the optimal duration for simple tactile pulse from a mobile phone RMV. The users were asked to rate vibrations of variable length. The ratings were the following: not detected, weak, moderate, strong, and too strong. They determined that a equilibrium between user comfort and detectability lies between 50 and 200 ms. This probably depends on several other variables also, but these results can be applied to mobile phone RMVs. Research on encoding time with temporal vibration pulse sequences could not be found. The Vibe Watch however is a real product which proves that the method must be viable in some way. Just how viable it is and if it can be implemented in a mobile phone will be investigated in this thesis. 3.3. Mobile Device Tactility Many mobile devices, such as phones, have a very simple Rotary Mass Vibrator (RMV). This is a cost effective solution but one of the cons is the lag induced by the spin up time and usually the lack of finer control. Motorola has used a Multi Function Transducer (MFT) which is a more unique approach. Both technologies can be used to produce the vibrations necessary for the application described in this thesis. Some mobile phones have more advanced vibration capabilities [Segan, 2005]. For example, there are models from Samsung called N330 and E770 which have VibeTonz [2006] technology from Immersion [2006]. The basic technology is RMV but it is more refined than usual. In addition to duration the intensity of vibration can be controlled. Also the electric motor can be spun backwards to create a different feeling. The 3rd edition of Symbian S60 has the necessary functions to control the speed and the direction of the RMV. The practical functionality of these controls was not studied nor used in this thesis. The functionality is implemented CHWRMVibra class of S60 3rd edition and the FP1 [CHWRMVibra, 2007]. 28 More refined control over the vibrating features might result in better recognition. For example, two different signals could be differentiated by amplitude in addition to their length. Some attributes could be simulated with a simple vibrating motor like Brown and Kaaresoja [2006] did in their experiments with roughness. On the other hand they got better recognition rates by encoding with intensity. Motorola has taken the MFT approach with some of their phones. It is basically a speaker (which is a voice coil, unless it is a piezoelectric speaker) which plays certain frequencies as vibration. Tactile effects can be felt when frequency is below 300 Hz and above that the user hears just sound. Chang and O'Sullivan studied if the users preferred the MFT enabled phone to the normal one and the results were positive for the MFT. However they used phones with radically different design so they recognize that the results are probably skewed because in addition to MFT there were other notable differences [Chang & O'Sullivan, 2005]. Poupyreve, Maruyama, and Rekimoto [2002] have presented some required attributes for an actuator to be used in a mobile device. First, it has to be as small as possible in order to fit in a wide range of small devices. Second, it must be lightweight and the actuator should not use a lot of power and the voltage should be low. Third, the actuator should be easy to integrate with a large number of devices. They have also defined human factor requirements. The actuator should have a fast response time since we can detect two stimuli to be consequent and separate with only 5 ms separating them. This is about five times faster than vision. The actuator should also be able to produce a wide range of frequencies as we can differentiate them too. Ideally it could produce frequencies from 1 to 1000 Hz. Also the intensity should be controllable. Finally, humans can also distinguish different wave shapes, so the actuator should provide that option too. At the moment there is no mobile handheld device, let alone a phone, which could provide all the versatility described above. The basic RMV is hopeless at best when faced with the above requirements. It only fulfills the technical requirements and flunks in all human factor requirements. MFTs fare a bit better, but the difference is small. 29 There have been some new approaches to the problem of the ideal actuator for mobile devices. Poupyrev and others [2002] used a novel actuator in their tests. It has some very desirable characteristics. The actuator is called TouchEngine and it is a kind of solid state muscle based on piezoelectricity which shrinks or expands depending on the polarity of the applied voltage. The actuator is very small and it can be battery operated. The latency is also minimal at around 5 ms. The actuator is so fast that amplitude and frequency can both be controlled. The downside is that the actuator displacement is only about 0.1mm so it requires some additional engineering to amplify the movement. 3.4. Summary In this chapter the applications and their background were discussed. Even though time is usually told by visual presentation, several other methods have existed for a long time. The need to tell time can be so grave that novel methods are employed when conventional methods fail to deliver. To invent new means to present time the researchers can draw inspiration from the history. At the moment time may be more important than ever before and there are countless ways to represent the time. It is everywhere with us on the screen of the mobile phone and in the corner of the computer display. However, there is still room for new methods to tell the time. There are several computer based applications which use vibrotactile cues to deliver messages. However, very few applications have been commercially deployed to consumers. Some of them may never be viable products but on the other hand this field is relatively young. A lot of work remains to be done in the field of actuator technology since the contemporary actuators leave a lot to be desired. Also new innovations are needed to create more useful applications. 30 4. A Method to Tell Time by Vibration There are two sections in this chapter: the description of the method to turn numerical time to temporal vibration sequences and the description of the application used in the experiments. The development of the method is also discussed and some alternatives are presented. The application itself is relatively simple so the description is also very straightforward. 4.1. From Numbers to Vibrations This method to tell time is based on reading time from temporal vibration sequences. The time is encoded in temporal patterns which the observer must decode. In other words: the time is encoded in time. The test device is a mobile phone with vibration capabilities and the program was written in J2ME. A phone was chosen because of the mobility. Devices which tell the time are usually mobile. When the information is conveyed by vibration, this is even more important as you have to touch the object. The application has some clear advantages. A software solution can be applied to any mobile phone with vibration capabilities. The perceived time can be interactively verified by entering it with the keypad ensuring correct time by the minute. The time could also be updated from the network automatically. The first prototype was made with a game pad equipped with an RMV. The game pad was similar to those in game consoles and it was controlled with the DirectInput API. The vibration frequency was around 10 Hz. This prototype proved that the user can sense the time by vibration with 80-100% accuracy even with the crude vibrations of the game pad. However this was only tested with two people. After this a J2ME application was implemented. It was tested and developed further in an iterative manner. However, the application is not guaranteed to work on all J2ME equipped mobile phones, as it uses relatively new extensions and the vibration controls have been implemented differently in different phones. As an afterthought, it might have been better to write the application in native Symbian, since it might have been able to provide finer control over the RMV [CHWRMVibra, 2007]. 31 In earlier versions some different methods of encoding were also used. One had three different signals. The additional signal encoded number five to make the whole sequence shorter. For example sixteen would have been encoded like: long-mediumshort. This method is somewhat similar to roman numerals. The sequences proved to be quite hard to recognize since the conversion task became more complicated with three different signals. It would probably require more training and be more susceptible to misconceptions as there would be three durations instead of just two. One version had an hour part from 0 to 23. This required very long break signals which were used to separate the hour and minute parts because both of them could be zero. Again there were three different signals instead of two and pilot tests showed that it complicated things. One option was to exclude single minutes altogether to make the minute part really simple and short. On the other hand it is often necessary to know the time by the minute so this was not done. The final coding for the time in the application is based on a telling a number at a time: 0 to 2 pulses for tens of hours, 0 to 4 pulses for single hours (zero only, if the first set has 1 or 2 pulses), a break, 0 to 5 pulses for tens of minutes, and 0 to 9 pulses for single minutes. The hours run from 1 to 24 so the first part before the break can not result in no pulses. There is at least one short pulse signaling 1 o'clock. The minute part can be zero. For example: 1 short pulse, break, 1 short pulse means that the time is 01:01. In some places the following shorthand is used: L for long, B for break and S for short. A long pulse was set to 600 ms and a short pulse to 100 ms. The pulses could be shorter, but the RMV in the phone needed enough time to spin up in order to produce sufficiently strong stimulus. This limited the minimum duration of the pulse. The pause between pulses was set to 1000 ms and the break between hours and minutes to 2000 ms. In the tests during development some people would have liked shorter pauses but when the pauses were cut shorter some complained that the pulses came too rapidly. The rotating eccentric mass in the phone has momentum and therefore the pulses tend to be a bit longer in reality than actually defined. Also the Java method sleep(int milliseconds) in the class Thread is not really accurate and might cause timing errors up to 200 ms or more. However, the premises are accurate enough for this purpose. When a pulse sequence was recorded with a capacitor microphone from the actual device the numbers were following: short pulse about 200 ms, long pulse about 800 ms, pause 32 about 1000 ms and the break between hours and minutes around 2500 ms. The vibration frequency could not be measured accurately, but the estimate was between 100 – 300 Hz. Also the frequency is probably the maximum the RMV in Nokia E50 [Nokia E50, 2007] can provide spin up time in shorter pulses taken into account. This recording is presented in Figure 2. These pulses and intervals are clearly distinguishable even though the actual values seem a bit erratic. The frequency or amplitude of the pulses can not be controlled as the vibration in the mobile phone supports only duration. The information transfer rate is below 1 bit per second (the application plays one binary value roughly every second), which is slow. A 24-hour clock can also be thought as two integer values: first hours with maximum value of 25 (maximum is 32, 24 used) and then minutes with maximum value of 26 (maximum is 64, 59 used) This results in a total of 11 bits. If it takes 10 seconds to play the time the transfer rate is 1.1 bits per second. Miller [1956] has presented an example where he defines that only 2.5 bits can be coded in different levels of pitch and 2.3 bits in levels of loudness. This gives some perspective about the amount of information we are encoding temporally. Figure 2: Pulse sequence recorded with a capacitor microphone from a vibrating mobile phone. The recording represents the time 07:12. When the user queries the time in a real situation he usually has some kind of rough estimate about it. This would help the interpretation in this application as well. For example, someone would usually know that when they wake up that the time is somewhere around 08:00 to 12:00. Also when they leave work they can assume that 33 time is usually past 16:00 for the rest of the day. This would narrow down the options when decoding the pulse sequence. 4.2. The Application The structure of the program is simple. At first, there is a simple form where the name of test subject among with some other information is entered. The form also includes the test setup variables such as the length of the sequences. The times can be either random or they can be randomized from an optional seed number. After these technical details are filled the actual testing interface is displayed. There are only two choices in two big buttons: answer and play. The user moves between the buttons by pressing left or right on the directional controller. A button is activated by pressing the enter key on the directional controller. The controls were designed and tested only on a Nokia E50 phone and all the tests were also made with that particular model. Name;0;22:47;22:47;CORRECT Name;1;10:48;10:48;CORRECT Name;2;21:04;21:01;WRONG Name;2;21:04;21:04;CORRECT Name;3;07:44;07:44;CORRECT Name;4;20:42;20:42;CORRECT Name;5;21:23;21:23;CORRECT Figure 3: Sample from the log the application generates Pressing the play button starts playing the time with a sequence of vibrating pulses. While playing the buttons are gray and if the user tries to enter something he will be prompted to wait. After the sequence has ended the user can enter the time with the keypad and then press the answer button. The focus is moved on the answer-button after the sequence has ended. If the answer is right, an auditory chime is played and the user can proceed to play the next sequence. The focus on the buttons is moved on the play-button. If the answer is wrong an auditory beep is played and the user is prompted to try again. The focus remains on the answer-button after a wrong answer. The user can play the pulse sequence again or try to answer straight away. After the user has answered correctly to all the time questions some statistics will be displayed and 34 pressing any key returns the user to the form described earlier or starts the next sequence of questions. All the correct and wrong answers are logged on a text file (see Figure 3). The user interface and progression through the program is depicted in Figure 4. Figure 4: VibroTime test application interface in an emulator Earlier the user was able to type the time with the keypad while the sequence was playing. This enabled user to offload some of the memory task by writing the hours on the keypad before the minutes started playing. This feature was disabled since it seemed more interesting to include this memory task. In reality there might not be such memory aids especially for the visually impaired. This means that the user has to keep the hour part in memory while the minutes are presented. This time varies from a few hundred milliseconds to as long as about 13 seconds depending on the amount of minutes. The application was also tested quickly without looking at the phone at all. Since the application plays auditory cues when the user answers, the application can be used without seeing the phone at all as long as there is some passive tactile feedback on the phone keypad to enable the user to enter numbers without seeing the phone. The application could have been modified to give vibrotactile feedback thus eliminating the aural channel completely, but that was not done in this version. In the first test without 35 visual information the user made typing errors on 3 out of 10 questions. After this a second set was tested and there were no errors at all. The real performance without visual cues might depend heavily on the design the keypad and the user's ability to use the keypad without seeing it. It is likely that this ability is better when the user is visually impaired. Even though this work is mainly theoretical and mostly springs from curiosity to know how accurately people can read the time from vibrating pulses alone, there might also be practical applications. Almost every mobile phone can produce vibration. This means that the application can be implemented with small software additions to almost any mobile phone or other small device with an RMV and a clock. In most situations visually impaired people can hear the time from a voice synthesizer, use a dedicated tactile watch, or a device with a braille display. On the other hand dedicated devices cost money. Also auditory signals are not discrete without headphones, which was also pointed out by van Erp [2002]. When the user wants to tell the time discreetly without looking at a display and with a common device, a vibrotactile pulse sequence from a mobile phone can be used. Also the perception of time can be tested by entering it on the keypad thus verifying the result. This might provide a better accuracy compared to a tactile clock face. The application presented here is simple and uses a common mobile phone. One additional advantage is the ubiquitous nature of the mobile phone: almost everyone has one and usually it is carried everywhere. It should also be studied if people would actually use vibration to read the time on their mobile phones if they had the need and the possibility. The design space for the application is straightforward. Classification used here was also used by Chang et al. [2002] in their ComTouch project. From the vibrotactile point of view the communication is uni-directional since the device only communicates by vibration. The data transfer is asynchronous as the user can either enter the answer or feel the time from vibration pulses. If the user could input numbers while receiving, the data transfer could be seen as synchronous. Input-output mapping system is asymmetric since output is vibration from the RMV and input device is the keypad. Data content is discrete. This means that the data (the time) has a finite number of possible values. 36 Poupyrev and others [2002] have divided tactile displays in two categories: direct tactile display and indirect tactile display. In direct displays single elements can produce tactile feedback themselves. For example when the user presses a button, the button itself may produce the stimulus. In indirect displays the stimulation comes from elsewhere. A mobile phone can be considered having an indirect tactile display since the whole device vibrates, not just the display. 37 5. User Experiments The goals and hypotheses of the two experiments along with test setups are described in this chapter. The tests were divided in two parts: Experiment I and Experiment II. In both experiments the task was to decode the vibrated time and then enter it with the keypad. Experiment I with five participants was aimed to evaluate the basic feasibility of the method. The Experiment II had ten participants and the sessions were longer and more data was gathered. The experiments show how the method performs in a controlled environment. The amount of test subjects is limited but the results should present some kind of information about time and vibrotactility. The effects of learning were also studied in Experiment II. 5.1 Goals and Hypotheses The first goal of the study was to determine how well the users could observe the time from a 24-hour clock presented with vibrotactile pulse sequences from a common nonmodified mobile phone. Tests related to the first goal are were conducted in the Experiment I. The second goal was to find out whether training and experience had any effect. Also the method needed further general testing. These goals where chased in Experiment II. It was determined in the pilot tests that the method was basically viable, but before the experiments several hypotheses were made according to previous information. According to pilot tests, the percentage of right answers would be around 80, which could get better if the subjects were trained. It was also deemed likely that every test subject would complete the sequences without any serious trouble. Also there would probably be more errors in the hour part since hours must be remembered when the minutes are playing, which is likely to distract the memory. Since the first experiment was limited, a second experiment was planned to further verify the method and gather more data. Another goal was to study the effects of learning associated with this method. The hypotheses was that nearly 100% overall recognition could be reached. Pilot tests showed that 90% overall recognition could be reached in an hour (only one error in 10 recognition tasks). Tests related with these 38 goals are called the Experiment II. The methodology is a simple empirical test with little intervention. User interactions are logged in a text file by the test application. Testing methodology is lightweight and designed so that it would be easy to gather test subjects, run the experiments, and finally analyze the results. The experiments could be made in any peaceful environment since the test application was ran in a mobile phone. Finally there were two separate experiments - one for each goal. The second experiment was a bit wider in scope and it was done six months after the first test. Also the testing methodology was changed a bit according to experiences from the first tests. The experiments are described in the following sections. 5.2 Experiment I The test setup in Experiment I was quite simple. The actual test consisted of 15 different sequences which all had to be answered correctly even if it would require several retries. All the actions were logged in a file which could be analyzed later. In the beginning of the tests the subjects filled in a simple form with their name, age, sex, and if they had any experience with vibrotactile devices. After this they were presented with a simple description of the time encoding method used in the tests. After they familiarized themselves with the description, they trained with a set of 10 random sequences. Every test subject answered to the same set of 15 time sequences in a random order. This way it may be possible to see if any particular time was hard to understand. It was not known which times were actually difficult so a random order was chosen. The users were allowed to replay the sequence if they were aware that they did not understand it at first. Replaying was not factored in the final results, only the wrong or right answers. This is because if the user knew he did not get the sequence it would not produce a wrong answer automatically as the user was consciously unsure. This is analogous to a situation where you recheck the time from your watch or mobile phone if you feel unsure about your perception. 39 The test environment was a room in a peaceful apartment with little distractions. Thus, the environment was not a laboratory, but the conditions were exactly the same for everyone and there was very little distraction. The test device was a Nokia E50 [2007] smartphone. The test setup could have been more strict and better controlled but since time and resources for this work were limited, these conditions had to do. The average age of test subjects was 24 years and the standard deviation was 2.08 years. None of them had any experience on vibration and time combined. They had experienced active tactile effects on their mobile phones when they alarmed or rang. Some had also used force feedback wheels, game pads, or joysticks. These experiences were not noted to provide any advantage in this experiment. All of them had normal eyesight and they had used a mobile phone before. Everyone had normal sense of touch by their own report. There were a total of 7 test subjects - 5 male and 2 female. After the test, the subject was asked to describe how he felt about the method. Despite the pilot testing, two flaws were also discovered during the testing. The first one was that the user could press the answer-button without entering a time. One user pressed the answer button since he thought he would be answering after the button press. Luckily this happened only once and the wrong answer was left out of results. The second flaw was that the users did not use any hearing prevention so they could hear the buzzing of the RMV in addition of feeling it. Both flaws were corrected in Experiment II. It seems that the test was largely unaffected despite these events. 5.3. Experiment II After it was determined that the method was viable it was decided that the effects of learning should be studied. It was presumed that the results would be below optimal at first and then improve with practice. If the test subjects reached around 80% overall recognition rate in Experiment I, how long would it take to reach 90% and how many would reach 100%? Even though effects of learning were briefly tested, Experiment II does not implement learning curve methodologies from cognitive psychology. These were also considered, but lack of time and resources favored a more straightforward and simple approach. 40 Experiment II included 5 new test subjects and 5 subjects who had also participated in Experiment I. Three of them were female. The average age of the subjects was 24.3 years and standard deviation of age was 1.89. The time between the tests was 6 months so the effects of learning from the previous tests should not be too strong. Nevertheless these two groups were handled separately. The software was changed very little from the Experiment I: logging features were enhanced and the users could not enter empty answers. One of the main changes was to log how many times the user played the pulse sequence describing a certain time. The user was allowed to replay the time before answering to better simulate a real world simulation where you can check the time again. Since other usability problems were not detected, the user interface was not changed. Also all the participants were instructed to wear hearing prevention equipment to mask the auditory cues from the RMV. It would have been even better if pink noise could have been played like Kaaresoja and Linjama did in their experiment [2005]. To prevent time-related risks on the experiment, maximum test time per subject was capped to 45 minutes. In the pilot tests 90% overall recognition rate was achieved in three consecutive runs within 45 minutes. This was also the requirement to pass the experiment. The sequence was shorter than in Experiment I. Instead of 15 different times, only 10 were asked per sequence. This change was made since the pilot tests for Experiment II showed that the test subject may get frustrated. The percentage of correct answers with one error would be 90% for 10 and 93% for 15. The second error would drop the percentage under 90% in both cases and the test subject would have to start from the beginning since three consequent runs with 90% performance was required to end the test. With 10 questions instead of 15, the sequences were completed faster and the subject lost less time, when he made the second error. So with 10 questions instead of 15, the “frustration factor” should be lower. Every test subject was asked the same overall set of times. The times were determined with a random number generator with a fixed seed. Overall recognition rate was determined by counting how many of the 10 questions were correct on the first try. For example 9 right answers on the first try out of 10 produces a recognition rate of 90% which is the target in these tests. To “pass” the Experiment II the test subject was required to make three consequent runs with 90% recognition. This condition was set to provide motivation for the test subjects to finish early. Hopefully the subjects saw this 41 as a challenge or a game and their motivation was better. However, as an afterthought, a more simple fixed length test without any termination conditions might have provided more information. The terminating condition was fulfilled surprisingly fast and therefore some users finished the test with only three runs (a total of 30 questions). In Experiment II the test subjects started the test without any training and other instructions were restricted to a simple instruction sheet and a basic oral introduction. No memorizing techniques were suggested whatsoever. Therefore the results should show how an untrained person with no relevant experience understands vibrotactile time signals. 42 6. Results The results from both experiments are discussed in this chapter. A detailed description of each test is given along with summary tables. In general, all the results were surprisingly positive. Most test subjects learned the method easily and reached high recognition rates. The tests prove that the method is feasible. However they do not tell if the users would be willing to use the method. 6.1 Results from Experiment I To clearly understand the results the raw data they are based on must be described. The application logged only wrong or correct answers. When the user entered the correct time the log states the requested time, the entered time and marks the line "CORRECT". When the user entered a wrong time the log states the requested time, the entered time and marks the line "WRONG". Overall performance is calculated by dividing the number of right answers with the number of total answers. This number indicates the ratio of right answers. Average answers to complete is the number of wrong or wright answers required to complete the 15 recognition tasks. Overall recognition rate was determined by counting how many of the 105 questions (7 subjects, 15 questions) were correct on the first try. The results have been summarized in Table 2. Attribute Result Number of test subjects 7 Number of time sequences asked per subject 15 Total questions asked 105 Overall recognition rate 80.0% Overall performance 78.4% Average answers to complete 19.14 Error quality (hours / minutes / both) Table 2: Summary of Experiment I 7 (24%) / 15 (52%) / 7 (24%) Everyone was asked to decode a series of 15 pulse sequences as a time value. The participants submitted 134 answers to a total of 105 questions, which is 105 right answers and 29 wrong answers. This means that the overall performance was 78.36%. Average answers to complete was 19.14 (mean) and the median was 18. Standard deviation is 3.48 and 71.4% of data is within one standard deviation from the mean. On 43 average the users made 4.14 errors. These are total errors where subsequent failed retries to answer are counted as separate errors. For example, making an error in two questions out of fifteen and getting it right on third time on both will result in 4 errors. The results are quite similar to hypotheses. The quality of errors was surprising. The hypothesis was that the errors would be mostly in the hour part of the time since it had to be remembered when the minutes were being played. The experiment revealed quite the opposite. There were over 50% errors in the minute part. From a total of 29 wrong answers seven errors were in the hour part, 15 errors in the minute part and seven errors where both parts were wrong. It might be that when the user is trying to keep the hour part in short term memory he can get confused with the more complicated minute part (1-24 vs. 0-59). Also the users thought that the hardest part was to remember the hours. The most difficult time seemed to be 10:48. It collected 8 errors. 02:55 followed with 4 errors and then 04:08 with 3 errors. After that most times had one or two errors. 11:58, 07:44 and 22:02 received no errors at all. 10:48 is being presented as L-B-L-L-LL-S-S-S-S-S-S-S-S. Failures to decode this sequence correctly might be due to hour part being so simple: one test subject said that he thought it was 01:48. The data shows that from eight errors, four were on the hour part and four on both. Overall recognition rate was 80%. This means that 80% of questions were answered right the first time. On average the users answered wrong the first time on three occasions during the 15 questions. As there were 4.14 errors per user counting subsequent retries the users usually got it right on the third try in one of the questions. This result is similar to the hypotheses. With some training users should be able to reach 90% recognition or better. This may be the value which can be vaguely compared with Brown and Kaaresoja's experiments on tactons and event recognition [Brown & Kaaresoja, 2006]. They shoved that a recognition rate of 72% was achieved with intensity encoding. Even better rates could be achieved by using length as an additional encoding attribute. However, it must be remembered that in the experiments described in this thesis the users were able to replay the time if they missed it the first time. 44 This experiment had a limited scope but it proved that the method is viable. Everyone learned how to use it with minimal training and completed the tasks quite easily. Error rates were positively encouraging since the users were almost untrained. Vibration from a single element can indeed be used to convey simple information. 6.2 Results from Experiment II The basic concepts in Experiment II are the same as in Experiment I described in the previous chapter. There are also some new concepts and values. One of them is completion speed which is the number of runs required to achieve 90% overall recognition rate three times in a row. Naturally, the minimum amount of runs is three. The concept of “passing the test” means that the user completed all three consequent runs with a recognition rate of 90% or more, this is also referred to as the terminating condition. In general, the results were again quite encouraging since 8 out of 10 subjects completed three runs in a row with a minimum recognition rate of 90%. Some of the subjects completed the task during the first three runs which is the optimal result. Unexpectedly one of the first time users answered right 30 times in a row beginning from the first question asked. Results from Experiment II were conclusive with the Experiment I in the way the individuals performed. The two worst performers in the Experiment I were also the two worst in Experiment II. It seems that this method of telling time has a really low learning curve and the differences in performance emerge from the personal attributes such as short term memory, concentration, and motivation. The results are summarized in Table 3. 45 Attribute Result Number of test subjects 10 (5 from Experiment I) Number of time sequences asked per subject 3 to 8 (10 questions per run) Total questions asked 504 Overall recognition rate 88.0% Overall performance 88.8% Average answers to complete (only those who passed) 4.38 runs (10 questions per run) Error quality (hours / minutes / both) Table 3: Summary of Experiment II 19 (29%) / 43 (66%) / 3 (5%) The error types followed the same pattern as in Experiment I. From total of 65 errors 29% were in the hour part, 66% in the minute part and 5% in both. In Experiment I the numbers were 24%, 52%, and 24%, correspondingly. In both tests the minute part had the most errors. Again, the users said that it was hard to remember the hours and that probably made the task of decoding the minutes even harder. There were minor differences between old and new test subjects in completion speed. The old subjects who “passed the test” in 45 minutes (two of them did not) had a completion speed of 43 questions and the new subjects had an average completion speed of 46 questions. The difference is not significant. The average number of right answers per run fit between 8.0 and 9.5. The averages start from round one as 8.8 and then fall to 8.7 and a peak of 9.5 is reached on the third run when 3 sets of 10 questions have been asked. The median was 8.75 and the standard deviation was 0.47. This means that the overall performance was quite high and there were no low numbers at all. The worst individual results in the whole experiment per run were 7 out of 10 right. Three out of ten subjects finished on the third run. Two of them had not used the method before and one of them had. Four subjects finished after the fifth run and one subject after the seventh. The two subjects who did not pass the test ran out of time on the fifth and eighth runs. Overall performance was 88.8% which was better than in Experiment I. The lowest performance was 84% and the highest 100%. The users replayed the time 1.10 times per answer on average. However, only two users from the users who passed were over the average with 1.42 and 1.20 plays per correct answer and all others were lower than 1.10. For those two who did not pass the numbers were 1.50 and 1.00. This experiment is too limited to really tell anything significant about the effects of replaying. Maybe it tells something about the motivation 46 for getting the right answer the first time. According to the data, the correlation between completion speed and replays is not significant (correlation co-efficient was -0.096). Then again, the data is limited and further studies are needed. The most difficult times of the clock in this experiment were 12:54, 14:47, and 17:27. They all collected four errors (total of 12). From the twelve errors 50% belonged to the two subjects who did not pass the test. Also they are about 18.5% of the total number of errors. Times which had 2 or more errors collected 52.3% percent of all errors. Since the number of errors was relatively small it would require much larger tests to really determine the difficult times. Also the results from Experiment I are not comparable with this one because of different times used and no correlations occurred. Therefore, the question of the most difficult time remains unresolved. It seems that the method is really easy to learn and after that the performance depends on the individual and their attributes. It seems logical when compared to Morse code which is more complicated and therefore has a more pronounced learning curve. It is possible, that with the right tests, longer sessions, and more subjects, some effects of learning could be observed. In the scope of these tests the individual differences seem to come up more easily, and the observations about the learning curve are lost in the noise. If the effects of learning could be observed in a larger experiment the theories about learning curve could also be adapted to them. It is possible that the short term memory could be trained to hold certain type of information better. This has been observed in Morse code users as they can hold a sequence of meaningless single letters in their short term memory while listening or feeling new ones at the same time [Tan et al., 1997]. 47 7. Discussion Even though the basic test procedures were pilot tested, two unintended features were discovered in the Experiment I after the tests had begun. In one case the user pressed the answer button before entering the time with the keypad. He thought he should first press the answer button and after that he could enter the time. The correct way is to enter the time first and then press answer. This was considered as a design flaw and the empty answers which were technically wrong were left out from the results. Luckily there was only one wrong answer because of this. Another flaw was that the users did not wear any equipment which would prevent them from hearing the residual sounds from the vibrating mobile phone. This means that the user also received some information from the auditory channel. At least the long pulses where the RMV has time to wind up to full speed will produce an audible buzz. After this was noted the experiment was conducted with two additional subjects. In these tests the users wore heavy duty hearing protection and could not hear any sound from the mobile phone. One subject achieved perfect recognition and the other made only one error. Wearing a hearing impairment device might even lead to better performance because it dampens all auditory distractions if there are any (like blind people have better aural performance). In Experiment II the user could not press answer until he had entered the time and the test subjects wore hearing impairment devices. In Experiment II the test ended if the user completed three consequent time series with a recognition rate of 90% or more (terminating condition). In the end, it felt that this requirement was unnecessarily strict. The experiment might have been better if the program would have monitored past 30 answers and if the recognition rate climbed to 90% or over, the test would have ended. Furthermore, the whole test end condition might have been unnecessary altogether. The decision to include such a condition was made to provide motivation for the test subjects. Also it was surprising how fast the users reached the requirements thus ending test for themselves. With all the information and experience gathered from the two experiments comes a desire to implement more and better tests. Sadly the time is limited and these two experiments have to do for this thesis. Also, even though it was consciously decided that the application used in the experiments should be kept simple, it gained a number of features during development. Less features might have lead to more simple test setups and the time spent 48 implementing this features could have used to make more and better tests. The general feedback from the users was positive. They found the method easier to learn than they first predicted. This was surprising since the method is quite unusual and no one had tried anything similar in the past. However, the results revealed that the learning curve may be quite flat. Some of them wished that the pulse sequence would be faster but then again they were not sure if they could really follow a faster sequence. Also, it was noted that high values in minutes make it more difficult to remember the hours. One user experienced an abnormality in the pulse sequence (two fast pulses with almost no gap) which was probably caused by a timing abnormality in thread processing. It is plausible since a mobile phone is not really a dependable real-time system. Users' willingness to use this method is a complicated question. When visual channel is available, most users are likely to use it. Also synthesized speech is probably preferred over vibration sequences if it is socially accepted or the user has headphones for private listening. People with no access to visual or auditory channels are left only with the option to feel the time. So, asking if people are willing to use the touch channel in situations where there are no other options depends heavily on their motivation. It is possible that the method is not very robust in a distractive environment. Memory capacity for vibration sequences seems small. This is logical since we communicate mostly by sight and hearing. Also, other senses may override touch in many situations. However, it would be interesting to test the method in a similar test setup like Vadas et. al. [2006] had when they tested reading speeds while the user walked on a path and listened or read text. This experiment depended heavily on the users short term memory in addition to perceiving vibration sequences. It is very likely that there would be a lot more correct answers if the users were able to enter down the hours right after they felt them. It would be interesting to see how fast the pulses could be played and to see how much distraction the user can take and still achieve the 90% overall performance. The signals could also be encoded with different tactile or even haptic feedback methods. The recognition rates would probably be better if spatial discrimination and/or waveform patterns were be used in addition to temporal encoding. Even intensity control could improve the recognition rate. However, this would require proprietary 49 devices with finer control than an off-the-shelf mobile phone provides. The Wang haptic clock [2007] used a different method to present the time by vibration. Although the method seems to be less accurate it could be compared with the method presented in this thesis. It would be interesting to study several different methods to present time with vibration and see if one of them would produce significantly better results. When compared to speech synthesis, the vibration method is likely to be inferior. This is because speech is an everyday method used to tell the time and it has a certain vocabulary with a known coding system. People have probably queried the time several hundreds or thousands of times during their lives. Modern speech synthesizers produce well formed speech and if one had to guess, the recognition rates for simple time information would be close to 100% from the beginning without any training. It is also worth noting, that time, as we usually understand it in the western civilization, is an abstract concept. The understanding of time is not something that people are born with. This may be realized when thinking about children who first learn the concept of hours and minutes. Then they need to learn how to read them from a numeric display and from an analog clock face. Earlier it was mentioned that some kind of rough knowledge about time could make the recognition task easier. For example if the person knew it was noon it would limit the possible interpretations. This is only an assumption and it would be an interesting to test this. 50 8. Summary It seems that the method described here is viable and people can tell the time by using it. 80% of all answers were correct already in the pilot test. When the users verified their perception of time by entering it on the keypad, everyone finished the Experiment I although for some it took more time. Also in real life situations the user could verify his observations by entering the time on the keypad. This would result in accurate recognition every time even though it might sometimes require retries. If the user has no visual or auditory channel in use, he can get the time reliably by using this method provided that the device has the correct time. Modern mobile phones can update their time automatically from the network which makes them quite trustworthy time tellers. The tests described in this thesis were more from the perspective of telling the time but it would be interesting to see what kind of results a more abstract test would yield. It would probably tell more about the basics of the whole method of conveying information. Also the cognitive processes of converting vibration to symbols like numbers could be studied further. Because the tasks experimented in this thesis seem to be more tied to individual performance than learning, similar methods could be used to measure concentration or maybe short term memory performance. The surprising thing in the experiments was that the hours were remembered quite well when listening to minutes. Errors made in the hour part only were only 24% of total errors (29% in Experiment II). In addition to the minute part being more complicated, partly concentrating on remembering the hours might make the task even harder. This thesis answers some simple questions, but it has raised a number of interesting questions for further research. If the users were tipped the rough measure of time such as morning or noon, would that improve the results? How much the ability to replay the time when the user is not sure of his perception improves the result? How would the results change when experimenting with different presentation speeds? What would be the most difficult time for the user to decode and why? Can a similar method be used to explore some individual attributes in people such as short term memory capacity and concentration? How would the results change if the users were distracted? How would the results change if there were ten times more test subjects? Does familiarity with mobile phones in general affect the results? How would a virtual tactile 51 clock work in a device with touch sensitive display and vibrotactile output? Due to the scope of this thesis these questions remain unanswered. The two main questions were answered. The first question was the following: can people understand time from vibration without any training? Yes they can. Both the Experiment I and the Experiment II show that with only a few introductory training sequences or with only a written instruction sheet the users can decode the time from the vibration. The second question was: how does training affect the results? It seems that training has little or none observable effect when the method is tested in the scope used in this thesis. The results did not improve dramatically over the time and the users who had tried the method before were not better than new users in the Experiment II. The results seem to tell more about the individual attributes of the users than the amount of training. The explanation probably is that the encoding scheme is very simple and therefore little learning is required to know how to decode it. On the other hand, the real time decoding task requires concentration and memory processing capabilities which brings up individual differences in results. It must be noted that the total amount of test subjects was only ten so the results should not be generalized too far. Finally, this thesis offers another view into the world of time and vibrotactility which seems to be somewhat unexplored. This is a small addition to the vast pool of studies about human-computer interaction using haptics, but the results of this study probably have some practical and theoretical use as well. 52 9. References [Big Ben, 1997] "Big Ben", A New Dictionary of Eponyms. Morton S. Freeman. Oxford University Press, 1997. Oxford Reference Online. http://www.oxfordreference.com/views/ENTRY.html? subview=Main&entry=t31.e29 (checked on 24th July, 2007) [Citizen, 2007] Citizen BL9009-54F. http://www.citizenwatch.jp/support/pdf/g900/e.pdf (checked on 23th July, 2007) [Brammer et al., 1993] Brammer, A. J., Piercy, J. E., Nohara, S., Nakamura, H., & Auger. Age-related changes in mechanoreceptor-specific vibrotactile thresholds for normal hands. The Journal of the Acoustical Society of America , 93, p. 2361. [Chang & O'Sullivan, 2005] Angela Chang and Connor O'Sullivan, Audio-Haptic Feedback in Mobile Phones. CHI 2005, April 2-7, Portland, Oregon, USA. [Chang et al, 2002] Angela Chang, Sile O’Modhrain, Rob Jacob, Eric Gunther, and Hiroshi Ishii, ComTouch: Design of a Vibrotactile Communication Device. [CHWRMVibra, 2007] CHWRMVibra Class Reference. http://forum.nokia.com/document/Cpp_Developers_Library/GUID-96C272CA-2 BED-4352-AE7C-E692B193EC06/html/classCHWRMVibra.html (checked on July 17th, 2007) [Classic Driver Magazine, 2007] The Ferrari 599 GTB Fiorano. http://www.classicdriver.com/uk/magazine/3300.asp?id=12863 (checked on July 16th, 2007) [Cooper, 1998] Richard Cooper, Visual Dominance and the Control of Action. Department of Psychology, Birkbeck College, University of London. [Brown & Kaaresoja, 2006] Lorna M. Brown and Topi Kaaresoja, Feel Who’s Talking: Using Tactons for Mobile Phone Alerts. CHI 2006, April 22–27, 2006, Montréal, Québec, Canada. [van Erp, 2002] Jan B.F. van Erp, Guidelines for the Use of Vibro-Tactile Displays in Human Computer Interaction. TNO Human Factors Soesterberg, The Netherlands 2002. [Frey, 2007] Martin Frey, CabBoots - Shoes with integrated Guidance System. TEI'07, 15-17 Feb 2007. [Fu & Li, 2005] Xiaoyan Fu and Dahai Li, Haptic Shoes: Representing Information By Vibration. IMAGEN Program, National ICT Australia. 53 [Geldard, 1957] Geldard F. A., Adventures in tactile literacy. American Psychologist, 12, 115-124. [Geldard & Sherrick, 1965] Geldard F. A., & Sherrick C. E., Multiple cutaneous stimulation: The discrimination of vibratory patterns. Journal of the Acoustical Society of America, 37, 797-801. [Gould et al., 1987] John D. Gould, Lizette Alfaro, Rich Finn, Brian Haupt, Angela Minuto, and Josiana Salaun, Why Reading Was Slower From CRT Displays Than From Paper. Proceedings of the SIGCHI/GI conference on Human factors in computing systems and graphics interface 1987, Toronto, Ontario, Canada April 05 - 09, 1987 [Ideo, 2002] Technojewelry for IDEO. http://www.ideo.com/portfolio/re.asp?x=50165 (checked on June 15th, 2007) [Iggo, 1987] Ainsley Iggo, Touch. The Oxford Companion to the Mind. Richard L. Gregory. Oxford University Press 1987. Oxford Reference Online. Oxford University Press. http://www.oxfordreference.com/views/ENTRY.html? subview=Main&entry=t159.e830-s3 (checked on 14th August, 2007) [Immersion, 2006] Immersion Corporation. http://www.immersion.com (checked on November 15th, 2006) [Kaaresoja & Linjama, 2005] Topi Kaaresoja & Jukka Linjama, Perception of Short Tactile Pulses Generated by a Vibration Motor in a Mobile Phone. Nokia Research Center and Nokia Technology Platforms, 2005. [Lederman, 1997] S.J. Lederman. Skin and Touch. Encyclopedia of Human Biology. Volume 8. (2nd edition, pp. 49-61). San Diego: Academic Press. [Loomis, 1981] Jack M. Loomis, On the tangibility of letters and braille. University of California, Santa Barbara. [Mazonne et al., 2003] Andrea Mazonne, Rui Zhang, and Andreas Kunz, Novel actuators for haptic displays based on electroactive polymers. Proceedings of the ACM symposium on Virtual reality software and technology, 2003, pp. 196-204. [Miller, 1956] George A. Miller, The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information. The Psychological Review, 1956, vol. 63, pp. 81-97. [Mills, 2003] David L. Mills, A Brief History of NTP Time: Confessions of an Internet Timekeeper. http://www.eecis.udel.edu/~mills/database/papers/history.pdf th (checked on July 23 , 2007) [Nokia E50, 2007] Nokia E50, http://business.nokia.fi/A4272027 (checked on August 21st, 2007) [Novint Falcon, 2007] Novint Falcon, http://home.novint.com/ (checked on October 18th, 2007) 54 [Noon Gun, 2005] Bianca Coleman, A visit to the Noon Gun is a blast. EsCape Times, December 07, 2005 Edition 1. http://www.capetimes.co.za/index.php? fSectionId=1860&fArticleId=3025538 (checked on July 23th, 2007) [Otaduy & Lin, 2005] Miguel A. Otaduy and Ming C. Lin, Introduction to Haptic Rendering. Department of Computer Science University of North Carolina at Chapel Hill [Painstation, 2007] Painstation. http://www.painstation.de (checked on July 16th, 2007) [Perez, 2001] Carlos Perez, Hammer and Gong. Timezone 31th Jan, 2001. http://www.timezone.com/library/cjrml/cjrml0016 (checked on 23th July, 2007) [Piezo, 2007] Basic Designs of Piezoelectric Positioning Drives/Systems. http://www.physikinstrumente.com/en/products/prdetail.php?sortnr=400800.00 (checked on October 15th, 2007) [Poupyrev, Maruyama, & Rekimoto, 2002] Ivan Poupyrev, Shigeaki Maruyama, and Jun Rekimoto, Ambient Touch: Designing Tactile Interfaces for Handheld Devices. Proceedings of the 15th annual ACM symposium on User interface software and technology [Rizzo & Messeri, 2004] Dominic Rizzo and Lisa Messeri, The Effect of Haptic Feedback in a Remote Grasping Situation. Department of Aeronautics and Astronautics Massachusetts Institute of Technology [RNIB, 2007] Royal National Institute of Blind People. http://www.rnib.org.uk/ (checked on June 16th, 2007) [Segan, 2005] Sascha Segan, Feeling the Next Generation of Vibration. PC Magazine 15.04.2005. http://www.pcmag.com/article2/0,1895,1786996,00.asp (checked on October 18th, 2007) [SensAble Phantom, 2007] SensAble Technologies, http://www.sensable.com/ (checked on November 7th, 2007) [Shaffer & Allison, 2007] Shaffer S.W. And Allison A. L., Aging of the somatosensory system: a translational perspective. College of Health Sciences, University of Kentucky [SMA, 2007] Shape Memory Alloys. http://www.cs.ualberta.ca/~database/MEMS/sma_mems/sma.htm (checked on October 18th, 2007) [Sherrick, 1991] Sherrick, C., Vibrotactile pattern perception: some findings and applications, in The Psychology of Touch, M. Heller and W. Schiff, Editors. 1991, Lawrence Erlbaum Associates. p. 189-217. 55 [Ship's Bell, 2007] "ship's bell",The Oxford Companion to Ships and the Sea. Ed. I. C. B. Dear and Peter Kemp. Oxford University Press, 2007. Oxford Reference Online. Oxford University Press. http://www.oxfordreference.com/views/ENTRY.html? subview=Main&entry=t225.e2193 (checked on July 24th, 2007) [Sidobre & Hayward, 2003] Daniel Sidobre and Robert Hayward, Calibrated measurement of the behaviour of mechanical junctions from micrometre to subnanometre scale: the friction force scanner, Meas. Sci. Technol. 15 (2004) 451–459. [Suzuki et al., 2002] Yuriko Suzuki, Minoru Kobayashi, and Satoshi Ishibashi, Design of Force Feedback Utilizing Air Pressure toward Untethered Human Interface. NTT Cyber Space Laboratories, NTT Corporation [Tan et al., 1997] Hong Z. Tan, Nathaniel I. Durlach, William M. Rabinowitz, Charlotte M. Reed, & Jonathan R. Santos. Reception of Morse Code Through Motional, Vibrotactile, and Auditory Stimulation. Perception & Psychophysics, Vol. 59, No. 7, pp. 1004-1017, 1997. [Tan et al., 1997b] Hong Z. Tan, Nathaniel I. Durlach, William M. Rabinowitz & Charlotte M. Reed, Information Transmission with a Multi-Finger Tactual Display. Scandinavian Audiology, Vol. 26, Suppl. 47, pp. 24-28, 1997. [Tarnow, 2005] Eugen Tarnow, The Short Term Memory Structure In State-Of-The Art Recall/Recognition Experiments of Rubin, Hinton and Wentzel. [Tissot, 2007] Tissot Watches. http://www.tissot.ch/ (checked on 23th July, 2007) [TouchSense, 2007] Immersion, TouchSense Technology for the Touchscreen Interface. http://www.immersion.com/industrial/docs/touchscreen_may06_V2LR.pdf (checked on January 16th, 2007) [Vadas et al., 2006] Kristin Vadas, Nirmal Patel, Kent Lyons, Thad Starner, and Julie Jacko, Reading on-the-go: a comparison of audio and hand-held displays, in Proceedings of the 8th conference on Human-computer interaction with mobile devices and services. [Verry, 1998] René Verry. Don't Take Touch for Granted: An Interview with Susan Lederman. Millikin University. [Vibetonz, 2006] Vibetonz. http://www.vibetonz.com (checked on November 15th, 2006). [VibeWatch, 2002] Vibe Watch Instructions. RNIB Customer Services, PO Box 173, Peterborough PE2 6WS. [VibeWatch, 2006] Vibe Watch MKII. http://onlineshop.rnib.org.uk/downloads/instructions/CV05.doc (checked on December 4th, 2006) 56 [Voice Coil, 2007] Voice Coil Theory of Operation, Pamela A. Eibeck and Brandon Muramatsu. http://bits.me.berkeley.edu/beam/vc_2.html (checked on October 18th, 2007). [Wang Haptic Clock, 2007] Haptic Clock. http://cwwang.com/wordpress/2007/05/24/haptic-clock/ (checked on June 28th, 2007) [Yamamoto et al., 2004] Akio Yamamoto, Shuichi Nagasawa, Hiroaki Yamamoto, and Toshiro Higuchitokyo, Electrostatic tactile display with thin film slider and its application to tactile tele-presentation systems. Proceedings of the ACM symposium on Virtual reality software and technology, Hong Kong, 2004, pp. 209 – 216. 57