Natural Web Interfaces

Editor: Munindar P. Singh • m.singh@ieee.org

Natural Interaction with

Visualization Systems
Robert St. Amant • North Carolina State University

As new input devices and interaction techniques emerge daily, how might they
improve interactive visualization?

T
echniques for information visualization, sci- sible, but none is entirely intuitive. Moving the
entific visualization, and visual analytics cutting plane means dragging the red slider, with
have reached the mainstream. It’s common to this control at a moderate distance from one of its
find scientists in domains ranging from archeol- effects. Notice that the plane’s vertical movement
ogy to zoology using interactive visualizations in also requires horizontal movement of the slider.
their work. Workers probe and analyze their data An arbitrary convention must be learned.
in the interests of business intelligence. End users These are simple, relatively small hindrances,
without special expertise can visualize data about but they can influence performance significantly
themselves — family history, finances, health and in interactions with 3D data. In human-computer
fitness, and so forth. interaction (HCI), these issues would be described
A great deal of interaction with data is online, in terms of integration of degrees of freedom
with the data, the analysis tools, or both being (low), spatial indirection (moderate), and direc-
maintained in the cloud. Demand for services to tional incompatibility (high).3 We can find com-
support interactive visualization and analysis will parable problems in information visualization, in
only increase with time, as well as with the growth the challenges of choosing subsets in large mul-
in size and complexity of data. While visual pre- tidimensional datasets, navigating through com-
sentations of data have grown more sophisticated plex networks of information, selecting targets in
over the decades, some have expressed concern cluttered presentations, and so forth. Research on
about interacting with the data. Niklas Elmqvist visualization environments has generally neglected
and his colleagues1 observe that interaction receives such issues, but now they’re seen as important, even
“little emphasis in visualization research,” and critical. Use of suboptimal or ineffective interaction
Bongshin Lee and her colleagues2 find that “com- techniques can be more than a missed opportunity;
paratively little has been done to take advantage of it can be a roadblock to progress.
the advances made in the interaction design com- A range of new input devices and types of inter-
munity.” What are the possible drawbacks? action have appeared in recent years. We might
To illustrate, in the context of scientific visu- imagine, for example, that a medium-scale mul-
alization, consider the interface to 3D Slicer, a titouch display, with some appropriate interaction
medical imaging system initially developed at the technique, would support more directness in manip-
Massachusetts Institute of Technology and Har- ulating images. Gestures in free space might offer
vard Medical School. Figure 1 shows a 3D repre- better integration: movement and rotation could be
sentation of a head, constructed with images from carried out in three dimensions by a user’s hand or
magnetic resonance imaging (MRI). The user can body, with the virtual image following suit. Soft-
rotate the head in three dimensions, and move the ware exists for such input to Web browsers today.
black axial cutting plane up or down. These oper- Kinected Browser from Microsoft, for example,
ations are carried out with a conventional mouse. makes skeleton tracking and depth information
This means that the two (linear) degrees of free- available as Document Object Model (DOM) events
dom of the mouse’s movement must be mapped and JavaScript structures. A library from Leap
to the three (rotational) degrees of freedom of the Motion relies on WebSockets to translate gesture
head’s movement; different mappings are pos- data into a form that a Web browser can handle.

60 Published by the IEEE Computer Society 1089-7801/15/$31.00 © 2015 IEEE IEEE INTERNET COMPUTING
 Natural Interaction with Visualization Systems

It’s important to realize, however,

that new input technology won’t by
itself lead to an improved, new gen-
eration of interactive visualization
tools. In early 2015, the editorial board
members of IEEE Computer Graphics
and Applications speculated about
future directions in computer graphics
and visualization.4 Themes included
the need to consider new platforms
for visualization (scaling up to very
large displays and scaling down to
mobile), new populations of users
(such as non-technical users inter-
ested in summaries of their personal
data), and new interaction techniques.
This last topic is my main concern in
this article. What might emerge from
a potential chaos of competing inter-
Figure 1. A three-dimensional representation of a head shown in 3D Slicer’s
action techniques? In the same CG&A
interface. Using a mouse, the user can rotate the head in three dimensions or
article, Joseph J. LaViola Jr. and Pak
move the black axial cutting plane up or down.
Chung Wong suggest that natural user
interfaces can provide an important
perspective.4 are not natural.”6 That is, gesture alone even the interaction techniques alone
isn’t enough. Users must understand don’t make an NUI. A badly designed
Natural User Interfaces which gestures are applicable in which interface that supports both speech and
Natural user interface, or NUI, is an situations, what a given gesture means, gesture might not be an NUI; it’s the
evolving term of art in HCI. That is, and how the system should respond characteristics of the interaction that
HCI researchers and practitioners use to a given gesture — users still need to make an NUI.
the term to mean specific kinds of learn and maintain a good conceptual A natural user interface is, perhaps
user interfaces, though there remains model of the system they’re using. surprisingly, an artificial thing, in par-
significant disagreement about what Daniel Widgor and Dennis Wixon ticular because gaining expertise in a
those interfaces are. give a more general account of NUIs in real-world activity typically requires
For Bill Gates, the Kinect is the key Brave NUI World, the most comprehen- hard work that might not be enjoyable
to NUIs. He writes, “[C]omputing devices sive account of NUIs to date. They write or automatic. Consider learning to play
will adapt to our needs and preferences that “a NUI fosters the development of the violin well, or learning photogra-
for the first time and humans will begin skilled behavior and engenders a feel- phy, or learning to use spreadsheet soft-
to use technology in whatever way is ing of mastery by eliciting appropriate ware. Virtuosos can be found in almost
most comfortable and natural for us,” actions from users and shaping them into any complex activity, and some are
and “with Kinect, we are seeing the skilled behavior smoothly, efficiently, naturals, having been accomplished as
impact when people can interact with and enjoyably.”7 By their account, an a novice and had little trouble making
technology in the same ways that they NUI isn’t a natural user interface but the transition to expert. The promise of
interact with each other.”5 Others have instead a natural user interface: one that an NUI is to reproduce this experience
taken up this view of NUIs, to the extent makes a user feel like a natural at what- for all users.
that in some circles “NUI” is synony- ever task is being carried out.
mous with “gesture-based interface.” Because the concepts are still rela- Visualization and Natural
It’s misleading, though, to think tively immature, it can be difficult to User Interfaces
that a NUI can be achieved simply by distinguish NUIs from other user inter- In the language of abstract visualization
using the Kinect as an input device, faces. Wigdor and Wixon identify three tasks, visualization systems are used
supporting gesture and spoken input, characteristics of NUIs: they’re enjoyable to select, explore, reconfigure, recode,
for visualization or any other complex to use; their use leads to skilled practice; abstract, filter, and connect bits of infor-
interactive task. Don Norman offers a and their use is appropriate to context. mation. The ways in which these tasks
blunt summary: “Natural user interfaces They also observe that the platform and are mapped to specific interactions with

november/december 2015 61
Natural Web Interfaces

a given visualization system determines Balakrishnan didn’t address learnabil- head around it better.”9 This captures
how natural the system is in use. Here, ity and improvement in performance the notion of feeling like a natural
I consider three interfaces that represent in the evaluation, they proposed that when first using a system.
natural interaction in different ways, individual users might define their The last illustration I want to discuss
to illustrate a few possibilities. None of own private gestures to be recognized is a proof-of-concept called CAPTIVE,
these systems is Web-based, but given by the system and used to execute a Cube with Augmented Physical Tools,
recent advances in Web technology, specialized tasks. We might further for exploration of 3D information (see
nothing stands in the way in principle. imagine guides and different types of Figure 2).10 The system hasn’t been
Mike Wu and Ravin Balakrishnan constraints to be implemented, such evaluated in user studies, but it dem-
developed their RoomPlanner system, that users might learn to recognize and onstrates promising directions for natu-
a 2003 prototype, to explore interac- distinguish better from worse layouts ral interaction. The CAPTIVE hardware
tion techniques on then-new table- more quickly and accurately. setup has three components: stereo-
top displays supporting collaboration RoomPlanner works in a very spe- scopic or monoscopic display, a haptic
and multitouch.8 With RoomPlanner, cialized domain and is more represen- pointing device, and a camera focused
two users sat facing each other across tative of early work in NUIs than of on the user’s hands. In the monoscopic
a DiamondTouch table, developing visualization systems. A more recent configuration, the user watches video
in their shared workspace the layout system, Jeffrey Browne and his col- of his or her hands on the monitor,
for furniture in an office. Some of the leagues’ SketchVis, works on domain- captured by a video camera mounted
interaction techniques tested in the independent data.9 SketchVis is a on its back; in the stereo configura-
RoomPlanner have come into common proof-of-concept visualization system tion, the user’s hands remain in front
use. For example, we can select an icon that relies on sketch-based interaction. of the monitor. In both configurations
representing a piece of furniture via a In many knowledge-intensive domains, the user holds a physical wireframe of
single tap and drag it to a specific loca- especially those that involve organizing thin wooden rods connected by colored
tion. We can use two fingers to rotate and interpreting large amounts of infor- plastic corners. Image-processing algo-
or scale a furniture icon, if the icon is mation, it’s common to develop initial, rithms recognize the corners so that
large enough to touch simultaneously prototypical examples of the informa- virtual objects can be projected inside
with two fingers. We can first select tion we would like to see, in forms that the cube, tracking its position and
a smaller icon with one finger, and can potentially highlight areas or fea- orientation. When the tip of the hap-
then rotate the icon around the point tures of interest. In a word, we sketch. tic pointing device is brought into the
of touch when another finger touches Sketching lends itself to creative display region, it’s visually augmented
the surface and moves. One user can thinking, and for many it’s enjoy- to reflect its function as a tool. Imag-
use a flick gesture to send furniture able and natural. As a precursor to ine a cloud of objects projected into the
icons across the table to the other user. constructing a visualization, analysts cube. Tools under development include
Other interaction techniques, seemingly might sketch out a frame for a data probes and prods for pointing at, select-
as general, are less familiar today. The chart, draw axes for specific attributes, ing, and moving the displayed objects;
entire room layout, furniture and all, and mark off sample points in a scat- magnifying and semantic lenses for fil-
can be rotated around its center if a ter plot. Depending on the semantics tering, recoding, and elaborating infor-
user lays one hand flat on the display of the data, the scatter plot might get mation about the objects; and cutting
and slides to one side or the other. The discarded in favor of a bar chart, with planes that produce slices or projection
authors even implemented a simple pri- additional data attributes brought in views of the objects.
vacy measure: a user can place the edge and new annotations added. SketchVis Interaction with CAPTIVE exploits
of one hand on the tabletop, blocking a supports this process, carried out with our experience with handheld objects
small region from the view of the other a stylus on a wall display or a tablet and tools. How hard is it to exam-
user across the table. In this hidden computer, by recognizing and infer- ine a virtual object from an arbitrary
region, the user can make small-scale ring chart properties and filling in data angle? It should be no more difficult
changes. automatically, as a chart is constructed. than examining an object, such as a
The RoomPlanner interface is In evaluation, Browne and his col- cube, held in the hand. Selecting a vir-
strongly suggestive of a natural user leagues found that SketchVis changed tual point in three dimensions can be
interface. Informal usability testing the nature of interaction with data difficult. Contrast this with touching
found that “participants required little for some participants. One novice in a point on a physical object. We can
practice to learn the gesture set and data exploration said, “I don’t spend brace one hand against the other, tak-
were able to use the gestures effectively a lot of time working with graphs, ing advantage of physical, mechanical
to create room layouts.” While Wu and and [SketchVis] feels like I can get my constraints. We can use both hands in

62 www.computer.org/internet/ IEEE INTERNET COMPUTING

 Natural Interaction with Visualization Systems

a coordinated fashion. We look directly

at our hands (the control) and the object
or material that they hold (the display),
because they’re naturally collocated.
CAPTIVE is an attempt to exploit these
aspects of interaction with the physical
world in virtual interaction.
With such systems moving out of
research labs into the real world, we
might ask about expected benefits. The
area is so new, however, that a body of
evaluation literature hasn’t accumu-
lated yet, either specifically for natu-
ral interaction with visualizations or
for NUIs in general. It’s possible to see
performance differences due to specific
interaction design choices, but whole-
system performance improvements that
can be attributed to the naturalness of
interaction remain a promise for the
future.

D espite the newness of NUIs, visu-

alization researchers and practitio-
ners find the approach interesting, even Figure 2. The CAPTIVE system (a Cube with Augmented Physical Tools)
exciting. How can they be constructed? combines a stereoscopic or monoscopic display, haptic pointing device, camera
Wigdor and Wixon lay out dozens focused on the user’s hands, VR imaging software, and real-world components
of design guidelines, from the very gen- to promote natural interaction with handheld objects and tools. Here, the user
eral to those that depend on the choice holds a physical wireframe of thin wooden rods connected by colored plastic
of specific interaction technology (such corners. Image-processing algorithms recognize the corners so that virtual
as a touch-sensitive surface versus objects can be projected inside the cube, tracking its position and orientation.
camera-based gesture recognition).7 As
a starting point, however, they identify
a few general rules to follow — rules directly, with actions understandable to dor and Wixon emphasize that relevant
that are reflected in and compiled from the novice because they arise in inter- context information be made clear to
experience with NUI systems. We can action with the real world. users (through, for example, visual rep-
translate these into a data visualization “Create an experience that, for expert resentations and feedback) so that they
context with little effort. users, can feel like an extension of their can develop a good conceptual model
“Create an experience that feels body.”7 We can imagine fully fleshed- of the system.
just as natural to a novice as it does out implementations of SketchVis and Lee and her colleagues describe
to an expert user.”7 It might be easi- CAPTIVE taking advantage of users’ NUI interaction as “a general interac-
est to appreciate such an experience in expertise — specifically, users with tion experience evoked on the person
contrast with tasks that don’t feel nat- experience in sketching or in the use of including instant feeling of success and
ural. Consider what’s needed to rotate tools that require fine motor control. If ease of use.”2 As a specific example
or scale a data view using a mouse: tasks could be designed to exploit our of such an experience, I think of the
choosing an origin, manipulating a motor as well as our cognitive skills, the CAPTIVE system as allowing users to
widget, entering and exiting a specific experience might feel more natural. hold and examine data literally in their
mode. An expert might carry out such “Build a user interface that considers hands. Naturalness in the interaction
tasks without thinking, but a complete context …”7 Context includes the physi- with visualization systems has obvi-
novice might struggle at first. With cal and social environments, the tasks ous appeal; it seems inevitable that
multitouch, as in RoomPlanner, we to be performed, users’ backgrounds, progress on natural user interfaces can
can carry out such tasks much more and the computing platform itself. Wig- improve interactive visualization.

november/december 2015 63
Natural Web Interfaces

Acknowledgment 5. B. Gates, “The Power of the Natural User Abstracts of the ACM Conf. Human Factors
The US National Science Foundation supported Interface,” Gates Notes, blog, 28 Oct. 2011; in Computing Systems, 2014, pp. 1315–1320.
this work through grant IIS-1420159. www.gatesnotes.com/About-Bill-Gates/
The-Power-of-the-Natural-User-Interface. Robert St. Amant is an associate professor in the
References 6. D.A. Norman, “Natural User Interfaces Are computer science department at North Caro-
1. N. Elmqvist et al., “Fluid Interaction for Infor- Not Natural,” Interactions, vol. 17, no. 3, lina State University. His research targets
mation Visualization,” Information Visual- 2010, pp. 6–10. models of interaction, drawing on concepts
ization, vol. 10, no. 4, 2011, pp. 327–340. 7. D. Wigdor and D. Wixon, Brave NUI World: in human-computer interaction, cognitive
2. B. Lee et al., “Beyond Mouse and Keyboard: Designing Natural User Interfaces for science, and artificial intelligence. St. Amant
Expanding Design Considerations for Infor- Touch and Gesture, Elsevier Science, 2011. has a PhD in computer science from the
mation Visualization Interactions,” IEEE 8. M. Wu and R. Balakrishnan, “Multi-Fin- University of Massachusetts. His popular sci-
Trans. Visualization and Computer Graphics, ger and Whole Hand Gestural Interaction ence book, Computing for Ordinary Mortals,
vol. 18, no. 12, 2012, pp. 2689–2698. Techniques for Multi-User Tabletop Dis- was published by Oxford University Press in
3. M. Beaudouin-Lafon, “Instrumental Inter- plays,” Proc. 16th Ann. ACM Symp. User 2012. Contact him at stamant@ncsu.edu.
action: An Interaction Model for Design- Interface Software and Technology, 2003,
ing Post-Wimp User Interfaces,” Proc. pp. 193–202.
ACM Conf. Human Factors in Computing 9. J. Browne et al., “Data Analysis on Inter-
Systems, ACM, 2000, pp. 446–453. active Whiteboards through Sketch-Based
4. L.M. Encarnação et al., “Future Directions Interaction,” Proc. ACM Int’l Conf. Inter-
in Computer Graphics and Visualization: active Tabletops and Surfaces, 2011, pp.
From CG&A’s Editorial Board,” IEEE Com- 154–157. Selected CS articles and columns
puter Graphics and Applications, vol. 35, 10. A. Chakraborty et al., “CAPTIVE: A Cube are also available for free at http://
no. 1, 2015, pp. 20–32. with Augmented Physical Tools,” Extended ComputingNow.computer.org.

From the analytical engine to the supercomputer,

from Pascal to von Neumann, from punched
cards to CD-ROMs—the IEEE Annals of the
History of Computing covers the breadth of
computer history. The quarterly publication
is an active center for the collection and
dissemination of information on historical
projects and organizations, oral history activities,
and international conferences.

www.computer.org/annals

64 www.computer.org/internet/ IEEE INTERNET COMPUTING