JythonMusic: An Environment for

Developing Interactive Music Systems

Bill Manaris, Pangur Brougham- Dana Hughes Andrew R. Brown
Cook Computer Science Dept. Queensland Conservatorium
Computer Science Dept. University of Colorado, Boulder, USA Griffith University, Australia
College of Charleston, USA dana.hughes@colorado.edu andrew.r.brown@griffith.edu.au
manarisb@cofc.edu,
broughamcookpj@g.cofc.edu

ABSTRACT wheel, etc.), controller type (e.g., mobile phone, motion

JythonMusic is a software environment for developing interactive sensor, glove, joystick, digital controller, glass sensor, etc.), and
musical experiences and systems. It is based on jMusic, a software computer software used. [3].
environment for computer-assisted composition, which was extended Several audio libraries and software packages for musical
within the last decade into a more comprehensive framework programming have been developed over the last two decades.
providing composers and software developers with libraries for In addition to manipulating audio, many focus on other
music making, image manipulation, building graphical user characteristics such as live coding, sample-level manipulation,
interfaces, and interacting with external devices via MIDI and OSC, or minimal programming requirements. Well-established
among others. This environment is free and open source. It is based packages include ChucK [6], SuperCollider [7], Pure Data [8],
on Python, therefore it provides more economical syntax relative to Csound [9], Minim [10], and Gibber [11].
Java- and C/C++-like languages. JythonMusic rests on top of Java, Most of the above packages assume programs are generated
so it provides access to the complete Java API and external Java- or executed in a console, and are not focused on simplifying
based libraries as needed. Also, it works seamlessly with other GUI development in addition to generating audio. Instead, GUI
software, such as PureData, Max/MSP, and Processing. The paper development either relies on packages available in the
provides an overview of important JythonMusic libraries related to underlying language (e.g., Java Swing), or a separate GUI
constructing interactive musical experiences. It demonstrates their development environment (e.g., Qt GUI used by SuperCollider,
scope and utility by summarizing several projects developed using Tcl/Tk used by Pure Data). These may require learning
JythonMusic, including interactive sound art installations, new relatively very complex APIs that expose needless low-level
interfaces for sound manipulation and spatialization, as well as components to the musician (e.g., the Java Swing package
various explorations on mapping among motion, gesture and music. contains well over 100 classes, including managers, events,
listeners, etc.), or possibly entirely new languages.
Alternatively, users may utilize existing front ends. For
Author Keywords instance, several front ends have been developed for ChucK
Music and interaction, user interfaces, interactive sound art and since its inception, specifically Audicle [12] and miniAudicle
installations, musical mapping, Python, software libraries [13, 14]. While these provide a more streamlined interface, and
useful visualizations, they still rely heavily on an embedded
CCS Concepts console for real-time coding, and do not provide a robust set of
• Applied computing → Sound and music computing; • Human- widgets for building GUI-based musical instruments. Third-
centered computing → Interaction design; • Information systems party GUI libraries are available for some environments (e.g.,
→ Open source software; for Processing, used by Minim), though these have varying
levels of complexity and maintenance.
Finally, modern web browsers allow building cross-platform
1. INTRODUCTION musical interfaces using web technologies (HTML5,
JavaScript, CSS, etc.), while leveraging the inherent layout and
Interactive computer music systems emerged in the last 40
networking capabilities of the browser. NexusUI provides an
years with the development of increasingly more powerful and
API for UI widget and sending OSC messages via UDP, as well
versatile I/O devices and software frameworks for interaction.
as builders for converting a Max/MSP patch and drag-and-drop
These advances in technology, as is usually the case through
interfaces [15]. Similarly, a simple browser-based API for
history, have been exploited in music to develop new and
creating and storing interfaces for the Gibberish.js audio library
innovative computer-based and electroacoustic music systems,
is presented in [16].
compositions, and interactive experiences [1].
This paper presents JythonMusic (http://jythonmusic.org), a
Rowe defines interactive computer music systems as music
software framework for developing interactive musical
systems that respond to human input, which gives them the
experiences. JythonMusic is an open-source environment with
ability to participate in live performances [2].
an extensive online API reference. It has been used to develop
Paine [3] extends this to create a more thorough taxonomy of
various interactive sound art and installations, among other
systems for interactive musical performance, or digital musical
projects, which explore various mapping strategies, and explore
interfaces (DMI). He takes into account existing musical
relationships among motion, gesture and music.1
instrument taxonomies, such as the Hornbostel and Sachs [4]
instrument taxonomy, consisting of aerophones, chordophones, 2. JythonMusic
idiophones, membranophones, and electrophones. He then JythonMusic is based on Python. It was originally developed
superimposes the dimensions specified by Birnbaum, et al. [5]. for teaching computer programming in a musical context [17],
Finally, Paine introduces additional dimensions to form a
conceptual map of musical interaction, including gesture type
(e.g., body movement, eye tracking, senstate surface, steering 1
This work has been funded in part by NSF DUE-1323605,
Licensed under a Creative Commons Attribution DUE-1044861, IIS-1049554, IIS-0849499 and IIS-0736480.
4.0 International License (CC BY 4.0). Copyright
remains with the author(s).
NIME’18, June 3-6, 2018, Blacksburg, Virginia, USA.

259
based on extensive earlier work in algorithmic music function assembles and plays a phrase once and then schedules itself
composition [18]. Within the last decade, it has grown to have to repeat.
extensive capabilities for developing music interfaces and Additional functionality is provided in the Mod class, which
interactive experiences across all dimensions mentioned in the contains a large selection of functions for transforming Phrases,
previous section. As a result, it may serve as a complement to Parts, and Scores. The View class contains functions to visually
existing music and other art-related software, such as PureData, display music in various formats. The music library also
Max/MSP and Processing, among many others. supports reading MIDI files as scores and writing scores to
JythonMusic is built on top of jMusic, an extensive music MIDI using the Read and Write classes respectively.
library for music composition and audio processing in Java Finally, multiple classes are provided for rendering sound.
[19]. It also incorporates other cross-platform libraries, such as The Play class plays Notes, Phrases, Parts, and Scores using
jSyn [20], and makes them available via Python syntax. MIDI. The AudioSample class plays / loops audio files, and
As an extension of jMusic, it supports computer-aided pitch shifts them in real-time. The LiveSample class can record
composition, generative music, instrument building, interactive sounds from a microphone, play / loop them, and pitch shift
performance and music analysis. It provides musical data them in real-time. The Metronome class synchronizes callback
structures (i.e., Note, Phrase, Part and Score), playback of functions, which supports live coding tasks, such as building
musical scores in real-time, as well as rendering in MIDI or complex rhythmic patterns.
audio for storage and later processing. It can also read and write
MIDI files, audio files and XML files, among others. As an 2.2 GUI Library
extension of jSyn, it provides a modular sound synthesis API The JythonMusic GUI library supports development of computer
based on unit generators, which can be combined to form musical instruments and graphical user interfaces, having an
complex timbres and software synthesizers. extensive set of graphics objects and widgets, with event-driven
All this functionality is now available through Python. This programming via callback functions, and various types of keyboard
design choice is very desirable – Python is a general-purpose and mouse events.
programming language designed to be succinct and easy to The main GUI object is a Display. A program’s GUI exists inside
read. Python programs tend to be about three times as short as a Display object (window). Displays contain other GUI components
equivalent programs in Java, and C/C++ [21]. Accordingly, (graphics objects and widgets). For example, this:
Python has become the most popular introductory programming
d = Display("Some Display", 400, 100)
language at US universities [22]. It is also used extensively by
companies such as Google. Finally, Python includes a large and
creates a 400x100 pixel window. No other code is needed, compared
comprehensive set of libraries for common algorithmic tasks.
to, say, Java Swing, or Tcl/Tk (among others), which require multiple
JythonMusic uses Jython, the version of Python running on
lines of sometimes cryptic code to set up, pack, and render a simple
top of the Java Virtual Machine. This gives it great portability,
window.
as it runs on all popular computing platforms. Additionally, it
provides access to the complete Java API through Python Similarly, creating displayable objects is as simple as creating an
syntax, as well as other external Java libraries, as needed. object, and then adding to the display – two lines of code. For most,
Finally, through relatively easy-to-use MIDI and OSC
libraries, JythonMusic works seamlessly with other music
software, such as Pd, Max/MSP, Ableton Live, as the examples
shown below demonstrate.
We believe JythonMusic provides a viable alternative to
existing systems (see Section 1.1), as it simplifies development
of interactive musical experiences. While perhaps not as
specialized as some of the above systems, to paraphrase Alan
Kay’s maxim, it makes simple things simple, and complex
things possible. JythonMusic comes with an editor, called
JEM, which encapsulates all available libraries, and provides
various useful keyboard shortcuts.

2.1 Music Library

The JythonMusic music library provides functionality to aid in
transcription, composition, and performance of musical works. It
supports both MIDI and audio material.
The Note class treats musical notes as objects, which require a
pitch and duration, with a thorough set of constants (such as C4, for
middle C, and QN for quarter note). In addition to standard MIDI
pitches, the music library supports microtones, for exploring non-
traditional tunings, as well as non-Western scales and ancient modes.
Note objects can be added in series to Phrase objects Phrases can
then be assembled inside Part objects, which allow instruments to be
set. Part objects can be assembled to create Score objects, which can
have a title and a tempo, among others. Each of these objects has
various setter / getter functions, which are documented online (see
http://jythonmusic.org/music-library).
Figure 1 demonstrates a code sample from a laptop orchestra Figure 1. A code excerpt using temporal recursion to
performance of Terry Riley’s In C (also see section 3.1). Here a perform Terry Riley’s In C. The JEM editor provides
temporal recursion pattern is being used, i.e., the loopMusic() various shortcuts for re-evaluating different sections of
running code.

260
objects, a single-line abbreviation exists. For example, the following 3.3 Time Jitters (2014)
adds a circle with a 10-pixel radius to the above display: Time Jitters [24] is a four-projector interactive installation (see
Figure 3), which was designed by Los Angeles-based visual
# x, y, and radius
c = Circle(200, 50, 10) artist Jody Zellen for the Halsey Institute of Contemporary Art
d.add(c) in Charleston, SC, USA.

The GUI library is implemented on top of Java Swing, so it provides

access to all Swing functionality. Natively, it offers the following
Graphics objects: Line, Circle, Point, Oval, Rectangle, Polygon, Arc,
Icon, and Label. Additional GUI control objects include: Button,
Checkbox, Slider, DropDownList, TextField, TextArea, and Menu
(top, and pop-up). Other widgets for building musical user interfaces
include HFader, VFader, Rotary, Toggle, Push, and XYPad.
Finally, the GUI library supports various keyboard and mouse
events (such as key press, mouse click, mouse move, and mouse
drag) to associate callback functions to handle these events. This
allows for developing elaborate, yet easy to comprehend interactive
behaviors. For example, Figure 2 displays the interface for a tunable Figure 3. Users interacting with Time Jitters at the Halsey
microtonal instrument, for exploring different tuning systems, based Institute of Contemporary Art on opening night.
on the ancient Greek tetrachord. For more information on the GUI
library, see http://jythonmusic.org/gui-library . Time Jitters includes two walls displaying video animation,
and two walls with interactive elements. The concept is to
create an immersive experience for participants, which
confronts them with a bombardment of visual and sound
elements. For more information, see http://bit.ly/timeJitters .

3.4 Diving into Infinity (2015)

Diving into Infinity [25] is a Kinect-based system which
explores ways to interactively navigate M.C. Escher’s works
involving infinite regression. It focuses on Print Gallery, an
intriguing, self-similar work created by M.C. Escher in 1956.
The interaction design allows a user to zoom in and out, as
well as rotate the image to reveal its self-similarity, by
navigating prerecorded video material. The system combines
JythonMusic with Processing using OSC. For more
information, see http://bit.ly/escherKinect .
Figure 2. GUI for a microtonal, tetrachord-based tuning
system. 3.5 Migrant (2015)
Migrant [26] is a cyclic piece combining data sonification,
3. SOME PROJECTS interactivity, and sound spatialization. It utilizes migration data
This section discusses a few more comprehensive projects collected over 23 years from 56,976 people across 545 US
developed with JythonMusic, which demonstrate its capabilities counties and 43 states. The piece was originally composed for
and potential for developing interactive musical experiences. Undomesticated, a public-art installation in the context of
They span various categories, including musical interaction, ArtFields 2015 (http://www.artfieldssc.org). It was performed,
interactive sound and visual art installations, new interfaces for as part of the ISMIR 2015 music program, in Oct. 2015, in
sound manipulation and spatialization, as well as various Málaga, Spain. The original is here http://bit.ly/migrant2015,
explorations on mapping among motion, gesture and music. and a more extended performance at the American College of
Greece here http://bit.ly/migrant2016 .
3.1 A Laptop Orchestra (2010)
In December 2010, we organized a student laptop orchestra 3.6 SoundMorpheus (2016)
performance. Students were introduced to the computer as a SoundMorpheus [27] is a sound spatialization and shaping
musical instrument and as a creative environment to develop interface, which allows the placement of sounds in space via
fluency with musical practices, including algorithmic arm movements. This system combines Myo armbands with
composition, developing simple computer instruments, and JythonMusic and PureData. For more information, see
composing exercises based on models in electroacoustic music http://bit.ly/soundmorpheus2 .
and minimalism [18]. Simultaneously, they were introduced to
programming in Python. For more information, see
http://bit.ly/charlestonLaptopOrchestra . 4. CONCLUSION
This paper provided a quick introduction to JythonMusic, a software
3.2 Monterey Mirror (2011) environment for developing interactive musical experiences and
Monterey Mirror [23] is an experiment in interactive music
systems and described some projects developed with it. Due to the
performance. It engages a human performer and a computer
limited space, only a subset of the available libraries were discussed.
(the mirror) in a game of playing, listening, and exchanging
The online website - http://jythonmusic.org - provides additional API
musical ideas. The computer player employs an interactive
documentation, as well as various code samples, videos, and other
stochastic music generator, which incorporates Markov models,
references. This environment supports computer-assisted
genetic algorithms, and power-law metrics. For more
composition, image manipulation, building graphical user interfaces,
information, see http://bit.ly/montereymirror .

261
and interacting with external devices via MIDI and OSC, among International Computer Music Conference (ICMC’04),
other features. (Miami, FL, USA, 2004).
[13] S. Salazar, G. Wang and P. Cook, miniAudicle and ChucK
5. ACKNOWLEDGMENTS Shell: New Interfaces for ChucK Development and
The following individuals have contributed to JythonMusic code Performance. In Proceedings of the 2006 International
development, API design and review, and testing: David Johnson, Computer Music Conference (ICMC’06), (New Orleans,
Paul Helling, Kyle Stewart, Margaret Marshall, William Blanchett, LA, USA, 2006).
Christopher Benson, Mallory Rourk, Seth Stoudenmier, and Kenneth [14] S. Salazar and G. Wang, miniAudicle for iPad
Hanson. The JEM editor is developed and maintained by Tobias Touchscreen-based Music Software Programming. In
Kohn. The jMusic library and materials have been developed by Proceedings of the 2014 International Computer Music
Andrew Brown, Andrew Sorensen, Rene Wooller, Tim Opie, Conference (ICMC’14), (Athens, Greece, Sep. 14-20,
Andrew Troedson and Adam Kirby. The jSyn environment is 2014), 686-691.
developed and maintained by Phil Burk. Chrestos Terzes provided [15] B. Taylor, J. Allison, W. Conlin, Y. Oh and D. Holmes,
invaluable information on ancient Greek music theory and the Simplified Expressive Mobile Development with
tetrachord. JythonMusic includes code partially supported by the US NexusUI, NexusUp and NexusDrop. In Proceedings of the
National Science Foundation (DUE-1323605, DUE-1044861, IIS- International Conference on New Interfaces for Musical
0736480, IIS-0849499 and IIS-1049554). Additional support has Expression (NIME’14), (London, UK, Jun. 30 – Jul. 4,
been provided by Google and IBM. 2014), 257-262.
[16] C. Roberts, M. Wright, J. Kuchera-Morin and T. Hollerer,
6. REFERENCES Rapid Creation and Publication of Digital Musical
[1] P. Modler. Interactive computer music systems and Instruments. In Proceedings of the International
concepts of Gestalt. In Leman M. (eds) Music, Gestalt, Conference on New Interfaces for Musical Expression
and Computing. Lecture Notes in Computer Science (NIME’14), (London, UK, June 30-July 4, 2014), 239-242.
(Lecture Notes in Artifical Intelligence), vol 1317. [17] B. Manaris and A. Brown. Making Music with Computers:
Springer, Berlin, Heidelberg, 1997, 482-494. Creative Programming in Python. Chapman & Hall/CRC
[2] R. Rowe. Interactive Music Systems: Machine Listening Textbooks in Computing - CRC Press, 2014.
and Composing. MIT Press, Cambridge, MA, 1992. [18] B. Manaris, B. Stevens, and A. R. Brown. JythonMusic: An
[3] G. Paine. Towards a Taxonomy of Realtime Interfaces for Environment for Teaching Algorithmic Music Composition,
Electronic Music Performance. In Proceedings of the 10th Dynamic Coding, and Musical Performativity. Journal of
Conference on New Interfaces for Musical Expression Music, Technology & Education 9, 1 (May 2016), 55-78.
(NIME 2010) (Sydney, Australia, June 15-18, 2010), 436- [19] A. R. Brown. Making Music with Java: An Introduction to
439. Computer Music, Java Programming, and the jMusic Library.
[4] E. M. von Hornbostel, and C. Sachs. Classification of Lulu, Raleigh, North Carolina, 2005.
Musical Instruments: Translated from the Original [20] P. Burk. JSyn - Audio Synthesis API for Java,
German by A. Baines and K. P. Wachsmann. The Galpin http://www.softsynth.com/jsyn (accessed Jan. 27, 2018).
Society Journal, 14 (1961), 3-29. [21] B. Manaris. Dropping CS Enrollments: or The Emperor's New
[5] D. Birnbaum, R. Fiebrink, J. Malloch, and M. M. Clothes?. ACM Inroads, 39, 4 (Dec. 2007), 6-10.
Wanderley, Towards a Dimension Space for Musical [22] P. Guo. Python is Now the Most Popular Introductory Teaching
Devices. In Proceedings of the 5th International Language at Top U.S. Universities. Communications of the
Conference on New Interfaces for Musical Expression ACM (July 2014).
(NIME’05), (Vancouver Canada, May 26-28 2005), 192- [23] B. Manaris, D. Hughes, Y. Vassilandonakis. Monterey
95. Mirror: An Experiment in Interactive Music Performance
[6] G. Wang, P. R. Cook and S. Salazer, ChucK: A Strongly Combining Evolutionary Computation and Zipf’s Law.
Timed Computer Music Language. Computer Music Evolutionary Intelligence 8, 1 (Mar. 2015), 23-35.
Journal 39, 4 (Winter, 2015) 10-29. [24] D. Johnson, B. Manaris, Y. Vassilandonakis, and S.
[7] J. McCartney, Rethinging the Computer Music Language: Stoudenmier. Kuatro: A Motion-Based Framework for
SuperCollider. Computer Music Journal 26, 4 (Winter, Interactive Music Installations. In Proceedings of the 40th
2002), 61-68. International Computer Music Conference (ICMC’14),
[8] M.S. Puckette, Pure Data: Another Integrated Computer (Athens, Greece, Sep. 14-20, 2014), 2014.
Music Environment. In Proceedings of the Second [25] B. Manaris, D. Johnson, and M. Rourk. Diving into
Intercollege Computer Music Concerts (Tachikawa, Japan, Infinity: A Motion-Based, Immersive Interface for M.C.
1996), 37-41. Escher’s Works. In Proceedings of the 21st International
[9] B. Vercoe and D. Ellis, Real-Time CSOUND: Software Symposium on Electronic Art (ISEA 2015), (Vancouver,
Synthesis with Sensing and Control. In Proceedings of the Canada, Aug. 14-19, 2015), 2015.
International Computer Music Conference (ICMC), [26] B. Manaris, and S. Stoudenmier. Specter: Combining
(Glasgow, Scotland, 1990), 209-211. Music Information Retrieval with Sound Spatialization. In
[10] J.A. Mills III, D.D. Fede and N. Brix, Music Programming Proceedings of the 16th International Conference on
in Minim. In Proceedings of the 2010 Conference on New Music Information Retrieval (ISMIR 2015), (Málaga,
Interfaces for Musical Expression (NIME’10), (Sydney, Spain, Oct. 26-30 2015), 2015.
Australia, June 15-18. 2010), 37-42. [27] C. Benson, B. Manaris, S. Stoudenmier, and T. Ward.
[11] C. Roberts and J. Kuchera-Morin, Gibber: Live Coding SoundMorpheus: A Myoelectric-Sensor Based Interface
Audio in the Browser. In Proceedings of the 2012 for Sound Spatialization and Shaping. In Proceedings of
International Computer Music Conference (ICMC’12), the 16th International Conference on New Interfaces for
(Ljubljana, Slovenia, 2012), 64-69. Musical Expression (NIME 2016), (Brisbane, Australia,
[12] G. Wang and P.R. Cook, The Audicle: A Context- Jul. 11-15, 2016), 2016.
Sensitive, On-the-fly Audio Programming
Environ/mentality, In Proceedings of the 2004

262