The Digital Baton: a Versatile Performance Instrument Teresa Marrin (1) Joseph Paradiso (2) (1) Media Laboratory, Massachusetts Institute of Technology marrin@media.mit.edu, http://marrin.www.media.mit.edu/people/marrin/ (2) Media Laboratory, Massachusetts Institute of Technology joep@media.mit.edu, http://physics.www.media.mit.edu/~joep Abstract The Digital Baton is a new electronic instrument which has been designed and built for the performance of computer music. The principle motivation for its design was to create a gestural controller which replicated as closely as possible the feel of a traditional conducting baton while retaining the maximum number of intuitive control parameters for the user. The Digital Baton contains several sensor systems which capture many of the modalities of hand motion and gesture for their application toward both discrete controls and continuous, expressive gestures. Current software for the Digital Baton makes use of its sensing functions in real-time music performance systems. 1 Introduction The Digital Baton is a new instrument for the performance of computer music, modeled after the function and form of a conductor’s baton. Its multimodal sensing systems measure fine position of the tip over a limited range, gross motion of the hand over infinite range, and finger pressure. It is an input device which is capable of a wide range of gesture-based control applications. The motivation for the construction of the Digital Baton came during the development of the Brain Opera project at MIT, which began in the summer of 1995. It was determined then that it would be necessary to have a control device, to be used during each performance, which could simultaneously execute a number of musical and coordination functions in real-time, including mouse-like pointing and clicking in three dimensions, generating beats for tempo control, and executing individual notes along with volume, envelope, and duration information. Initially, these functions were designed into a threedimensional electric field sensing system similar to that implemented in the “Sensor Chair”[6], but this was determined to be both too complex to build and debug in the given time, and not optimal for sending discrete, timed controls. The authors decided that a hand-held device would more accurately sense quick, articulated motions of the small motor system in the hand, and provide a more accurate representation of the position and motion of a hand in space without too much processing overhead. Figure 1. The Digital Baton in March 1996, soon after its initial construction; three of the pressure sensors are visible under the translucent, compressible surface of the handle. (photo by Chris Verplaetse) The authors also felt that such a device should replicate, to a reasonable extent, the shape and feel of a traditional conducting baton, so as to be held and used in a way familiar to trained conductors. Some departures from this intention were taken, however, in order to include as many degrees of freedom as possible. The result was a device of about 220 grams in weight, with a compressible surface, enlarged handle, balanced ergonomic form, and eleven degrees of freedom. This device, of which only one copy was made, has been used in over one hundred and thirty performances, with some modifications but no hardware failures to date. 2 Previous Work Numerous other baton-like interfaces and software applications have been developed over the years; their contributions have been studied and acknowledged by the authors. A few related projects are discussed below for their relevance to the conception and implementation of the Digital Baton. 2.1 Max Mathews, “Radio Baton” In the 1970s, Professor Max Mathews determined that human performance gestures could be roughly approximated as functions of motion over time at a sampling rate of 200 hertz; this became the basis for the adoption of the MIDI standard. His program, entitled “Conduct,” allowed a person to control musical effects such as amplitude, tempo, and balance over the course of an entire piece of music. This was among the first known attempts to give conducting-like control to a human user.[7] More recently, Professor Mathews created a device called the "Radio Baton," which uses a coordinate system of radio receivers to determine its position. The array of receivers sends its position values to a control computer, which then sends commands for performing the score to the ‘Conduct’ program. Approximately twenty prototype copies of the Radio Baton exist at various computer music centers around the world. No commercial version exists yet, although a production version is currently being designed at Marian Systems in Lafayette, California. 2.2 Keane, Smecca, and Wood, “MIDI Baton” Developed in 1990 by David Keane, Gino Smecca, and Kevin Wood at Queen's University in Canada, the ‘MIDI Baton’ was a hand-held electronic conducting system. It consisted of a brass tube which contained a simple handmade accelerometer, connected to a belt pack unit with an AM transmitter and two switches (‘stop/continue’ and ‘reset’). The belt-pack transmitted three channels of information (data from the accelerometer and switches) to an AM receiver. A microprocessor then decoded the beat and switch information, translated it into a MIDI-like code, and sent that code to command sequencing software on a computer.[7] The system was operated by holding the baton and making beat-like gestures in the air; the beats were used to control the tempo of a MIDI score. 2.3 Yamaha, “Miburi” Another unique hand-held musical instrument is the “Miburi,” developed by the Yamaha Corporation and released commercially in Japan in 1994. The Miburi system consists of two small keypads with eight buttons (one keypad for each hand), a vest embedded with six flex sensors, a belt-pack, and a wireless connection to an external synthesizer unit. The result is a lightweight, distributed network of small devices which can be worn over or under clothing. The Miburi has many merits, including accurate flex-sensing mechanisms, a wide range of sounds, and an enormous amount of discrete control to the user. It is also particularly effective when combined with other Miburis into small ensembles, in the same way that traditional chamber ensembles or rock bands work. 2.4 Jan Borchers, “WorldBeat” In 1995, Jan Borchers [1] developed a novel set of applications for a baton-like interface to be used in an installation for the general public. His approach was to use a commercially-available device -- in his case, the “Lightning” system by Don Buchla -- and focus on the interaction design and usability issues for an experience of a few minutes by a user of arbitrary background. His interactive music applications, written in Max, span a range of interesting tasks, including jamming on a piano along with a jazz sequence, generating beats in order to conduct the tempo of a classical score, and improvising in a Theremin-like mapping of one hand to pitch, and one hand to volume. In most cases, his applications require the use of two “Lightning” devices; one in each hand. 3 Design Concept Given the fact that several successful conducting instruments and programs have been developed before, the design goals for the Digital Baton were somewhat specific: to capture as many parameters of motion of the right hand as possible with a well-balanced object that would allow the user to gesture with it as if with a traditional conducting baton. Its shape was intended to allude to its use as a musical controller for conducting, but also to allow for unimagined future applications. The initial design constraints decided upon for the Digital Baton were that it should be easy to hold in one hand without a strap, shaped in such a way that at least four fingers could be articulated separately on its surface, and squeezable. It was also determined that it should have the capability of pointing and should measure both absolute position and orientation. Wirelessness was deemed valuable, but was considered less crucial than keeping the overall size and weight as low as possible. Investigations into the shape and size of the baton were begun by molding different clay-like materials into various shapes and gesturing with them to discover the optimal placement of the fingers and palm and wrist in relation to each other. Initial designs favored a tubular shape, but this was discarded after initial tests showed that it forced the wrist into a fixed position and didn’t allow for flexibility in finger placement. A more bulbous, tear-drop-like shape was adopted when it became clear that it allowed for opposable motion between the fingers, was easily graspable in the palm, and allowed for the wrist to easily rotate and point in any desired direction. The primary axis of the baton was then aligned with the index finger, so as to enhance the ease of use. 4 Physical Implementation In addition to considerations of shape and use, considerable time was spent on the design of the sensor suite both on-board the baton itself and external to it. Joseph Paradiso built or integrated most of the components, with consideration for anticipated user needs, expense, and portability. 4.1 Sensor design The hardware in the Digital Baton system consists of a molded baton unit, an external infrared sensor, an external tracking unit, a computer, and various MIDIcontrollable synthesizers and samplers. The sensors on the baton include an infrared LED, five piezo-resistive strips for finger and palm pressure, and three orthogonally-placed accelerometers. The infrared LED and sensor are used for tracking the position of the tip, providing absolute 2D position and a third value for the intensity of the signal (which provides some measure of the horizontal distance from the tip of the baton to the sensor). The accelerometers provide information on the acceleration of the baton handle along three axes, as well as information which can be filtered to detect beats and relative orientation of the baton. The five piezo-resistive strips are used as pressure sensors over five regions of the surface, and capture pressure data from four fingers and the palm. The infrared LED is situated at the tip of the baton, and modulated at 20 Khz. The infrared sensor sends a separate data stream to the tracker unit, consisting of two absolute position values, one intensity value and one signal-quality value. The baton unit sends its own data stream via cable to the tracker, including values for 3-axis acceleration, 3-axis orientation, and surface pressure. The tracking unit converts and sends the Figure 2. Schematic diagram for the sensor systems in the Digital Baton. signals to the computer, as well as displaying many of them for diagnostic use via meters on its front panel. The body of the Digital Baton consists of a clear tube attached to a clear urethane base into which the piezoresistive strips have been molded. Underneath the pliable surface of the base is a hollow, hard shell which houses the more delicate electronics, including the accelerometers, a PIC and a multiplexer. The PIC samples signals from the piezo-resistive strips and accelerometers. The infrared sensor consists of a camera housing a position-sensitive photodiode; this camera is only sensitive to the 20 kHz signal emitted from the infrared LED on the tip of the baton; all other light sources are ignored. The photodiode in the camera directly produces a signal that determines the horizontal and vertical coordinates of the baton tip; no video processing is required. A bright red LED is also potted into the baton body near the tip; the intensity of this LED can be varied under MIDI control for visually expressive purposes. 5 Evaluation and Conclusions The Digital Baton did not function in all the ways it was originally intended to; for example, the authors did not anticipate the extent to which it would be "played" like a traditional instrument by combining finger pressure with 2D position in order chose notes by pointing and triggering, or shaping them with finger pressure. It was also necessary to modify the baton a few times; the first modification took place very soon after it was built, when it was discovered that the carefully molded urethane surface interfered with the resolution of the pressure sensor under the surface. This was problem was fixed by shaving off the top 5 mm around the whole base, which destroyed the carefully-molded exterior surface but improved the pressure sensitivity. One unpleasant discovery included the realization that the wire from the baton would occasionally get caught underfoot and cause the baton to be dropped. This issue has not yet been addressed, since it is not clear whether an untethered baton would be dropped less frequently. The primary contributions of the Digital Baton, however, have been positive: it represents significant new work in the design of performance interfaces as a digital object which combines some of the tactile responsiveness of a traditional musical instrument with ergonomic design, and quantity and resolution of sensory data channels. In conjunction with software systems (both existant and in development), it can be a powerful and versatile tool for new work at the intersection of music and gesture. 6 Acknowledgments The authors would like to thank Professor Tod Machover and the Hyperinstruments research group at the MIT Media Laboratory for their support and guidance in the development of the Digital Baton. We would also like to thank Maggie Orth, Chris Verplaetse, Ben Denckla, Patrick Pelletier, and Pete Rice for their contributions to the project. Finally, we would like to express our gratitude to David Morgan of Motorola for his many kindnesses and encouragements during the course of this project. References [1] Borchers, Jan. “WorldBeat: Designing A BatonBased Interface for an Interactive Music Exhibit.” CHI ‘97 Conference Proceedings, pps. 131-138. [2] Darrell, Trevor and Alex. P. Pentland, "Space Time Gestures," Proceedings of the IEEE Conference, CVPR, 1993. [3] Marrin, Teresa. “Possibilities for the Digital Baton as a General-Purpose Gestural Interface.” CHI ‘97 Extended Abstracts, pps. 311-312. [4] Morita, Hideyuki, Shuji Hashimoto, and Sadamu Ohteru. "A Computer Music System that Follows a Human Conductor." Computer Magazine, vol. 24, no. 7, July 1991, pp. 44-53. [5] Paradiso, Joseph. "New Instruments and Gestural Sensors for Interactive Music Performance," to be published. [6] Paradiso, Joseph and Neil Gershenfeld. “Musical Applications of Electric Field Sensing,” to appear in Computer Music Journal. [7] Roads, Curtis. The Music Machine. Cambridge, Massachusetts: The M.I.T. Press, 1989. [8] Sawada, Hideyuki, Shin’ya Ohkura, and Shuji Hashimoto. “Gesture Analysis Using 3D Acceleration Sensor for Music Control.” Proceedings of the International Computer Music Conference, 1995. San Francisco: Computer Music Association, 1995, pp. 257260.