Keywords

1 Introduction

Our homes are becoming increasingly interactive. Small household appliances, like lighting controls and thermostats, are being enriched with new interaction modalities. Users can now interact with their household environments using their smartphones, speech commands and even in-air gestures. Gesture interaction is especially compelling as users can interact from across the room and without having to first locate their smartphone or a remote control. Unlike speech, gestures do not disturb others and work in noisy household environments, where chatter, music and television may cause speech recognition difficulties.

In-air gesture interaction, like speech, can be uncertain and users require lots of feedback during interaction, so that they can gesture confidently and effectively. Many household appliances have limited ability to give feedback, however. They typically have small displays, or no display at all, meaning visual feedback would be difficult to see from a short distance away. Users would lose the benefits of gesturing from across the room if they had to approach a device first to see feedback about their gestures and the effects they are having.

Changing the form factor of such devices to accommodate a larger screen is undesirable, as doing so increases size, cost and power demands. In our research, we are exploring other ways for household controls to give users feedback during interaction. This paper describes the Gesture Thermostat, a gesture-controlled thermostat dial which gives users multimodal feedback about their gestures. Interactive lighting illuminates the surrounding wall to give visual feedback as users gesture (see Fig. 1). We also use sound, from within the device, and vibration, from a smart-watch, to enhance the feedback.

2 Related Work

Many have found that users are more willing to interact with small devices from a distance than they are to approach them for input. Valkkynen et al. [7] found that users preferred to interact with objects by pointing at them, rather than touching them, when more than a step away. Even when on their feet or walking past objects, users would rather gesture than go out of their way to touch them. Rukzio et al. [5] also found that distal interactions were more preferred than approaching objects to interact with them. These distal interactions allow what Koskela et al. [2] call instant control of household appliances: immediate, convenient access to basic functionality.

Offermans et al. [3] looked at interaction with household lighting. They found that access to controls had a strong influence on willingness to interact: users were more likely to interact when controls were easily accessible. They noted that mobile devices – remote controls and smartphones – often required too much effort and had inconsistent availability. We think in-air gestures could be a convenient alternative, allowing direct interaction when desired. Users could gesture from across the room for instant control of basic functionality, like switching lights off or setting the air conditioning to a cooler temperature.

In-air gestures are also available when other interaction modes are inconvenient or unavailable; for example, touch is less appropriate when cooking. Kinect in the Kitchen [4] explored this scenario further, allowing users to browse recipes and set cooking timers using in-air gestures. They found that gestures were effective but users required more feedback. Others have used in-air gestures for interacting with lighting controls, [1] and [6]. Both used the lighting itself as functional feedback: users saw immediate feedback from how the lights changed in response to gestures. Functional feedback is often unavailable and gives limited insight into interaction, however. Users receive little feedback during gestures and if movements are unrecognised, they receive none at all.

3 Gesture Thermostat

Gestures give users another way to interact with household devices, although they need sufficient feedback to help them interact. Many household appliances have limited display capabilities, which limits their ability to give feedback. Functional feedback is often unavailable and does not give users enough information to help them gesture. We are exploring other ways of giving users feedback during gesture interaction with household devices. We focus on interaction with thermostats as these are common devices with limited displays and no immediately noticeable effects; such devices benefit most from extra feedback.

Our Gesture Thermostat prototype (see Fig. 1) has three types of output which can be used for gesture feedback: interactive light, sound and vibration. Lights embedded in the thermostat dial (46 LEDs, approx 7 mm apart) illuminate the surrounding wall, giving visual feedback over a large area which users can easily see from across the room. An internal speaker provides audio feedback. Wearable devices, like smart-watches, could also be used as tactile displays for feedback. We use a simple actuator as a prototype smart-watch. In the rest of this paper and in our poster, we describe the Gesture Thermostat, its gestures and feedback designs, and initial findings from a user evaluation.

Fig. 1.
figure 1

The Gesture Thermostat illuminates surrounding areas for visual feedback (Color figure online).

Full size image

3.1 Gesture and Feedback Design

We use gestures for basic functionality – adjusting temperature – as this allows instant control [2]. More complex interactions, like programming the thermostat schedule, are less suited for in-air gestures. We give users two ways of adjusting temperature: (1) imprecise “punch” gestures, which increase or decrease temperature by a few degrees; and (2) precise “dial” gestures, which allow finer-grained temperature control.

Users initiate interaction by raising their hand with a closed fist. Upon detecting this gesture, the thermostat plays a short tone, emits a short vibration if a smart-watch is available, and reads the current temperature setting aloud; for example “twenty-two degrees”. All lights turn on with a low brightness and pulse slowly to show continued responsiveness. A single area of coloured light shows the temperature setting; blue light for cooler temperatures on the left (Fig. 1, left), red light for warmer temperatures on the right (Fig. 1, right).

From this position, users can quickly move their hand up-then-down or down-then-up to raise or lower the temperature, respectively. Audio and tactile feedback are given to confirm a recognised gesture and lights update to show the new thermostat setting. Alternatively, users can open their hand, as though about to grasp the dial, and turn it left or right to lower or raise the temperature. We use rate-based rather than position-based control due to hand instability and sensing inaccuracy. As users perform the dial gestures, white lights stop pulsing and the coloured area of light pulses instead; this helps users identify between interaction modes. Users are given feedback about their gestures, as before.

3.2 Initial Evaluation Results

Our initial evaluation asked eight users to complete tasks using the thermostat. Our findings suggest that our feedback designs support users effectively. Evaluation participants generally understood how feedback related to their hand movements and felt it made the device seem responsive; however, some wanted more information, especially about gesture sensing. When users encountered tracking issues, they were aware that something was wrong, but did not know why. More feedback about sensing could help users overcome these issues.

Although light feedback was effective on its own, users found it most salient and useful when paired with audio and tactile feedback. Light feedback was continuous which helped show responsiveness, while discrete audio and tactile feedback were easily noticed. Participants suggested that non-visual cues made visual changes more noticeable and made the thermostat seem more responsive.

4 Summary

In-air gestures allow users to interact with small household devices quickly and from a distance. Users need feedback to gesture effectively, although many household devices have limited feedback capabilities. Our Gesture Thermostat exemplifies how embedded lights, sound and vibration from wearables can be used to give gesture feedback. Our initial findings are positive: these outputs help users gesture effectively and make devices fun and responsive.