Article

Authoring sensor-based interactions by demonstration with direct manipulation and pattern recognition

Authors:

Björn Hartmann,

Scott R. KlemmerAuthors Info & Claims

CHI '07: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Pages 145 - 154

https://doi.org/10.1145/1240624.1240646

Published: 29 April 2007 Publication History

Abstract

Sensors are becoming increasingly important in interaction design. Authoring a sensor-based interaction comprises three steps: choosing and connecting the appropriate hardware, creating application logic, and specifying the relationship between sensor values and application logic. Recent research has successfully addressed the first two issues. However, linking sensor input data to application logic remains an exercise in patience and trial-and-error testing for most designers. This paper introduces techniques for authoring sensor-based interactions by demonstration. A combination of direct manipulation and pattern recognition techniques enables designers to control how demonstrated examples are generalized to interaction rules. This approach emphasizes design exploration by enabling very rapid iterative demonstrate-edit-review cycles. This paper describes the manifestation of these techniques in a design tool, Exemplar, and presents evaluations through a first-use lab study and a theoretical analysis using the Cognitive Dimensions of Notation framework.

References

[1]

Arduino Physical Computing Platform, 2006. http://www.arduino.cc

[2]

LabView, 2006. National Instruments. http://www.ni.com/labview

[3]

SWT: The Standard Widget Toolkit, 2006. The Eclipse Foundation. http://www.eclipse.org/swt

[4]

Virtual Studio Technology (VST), 2006. Steinberg. http://steinbergcanada.com/technology/vst.htm

[5]

Andreae, P., Justified Generalization: Acquiring Procedures from Examples, Unpublished PhD Thesis, MIT, Dept. of Electrical Engineering and Computer Science, 1985.

Digital Library

[6]

Avrahami, D. and S. E. Hudson. Forming interactivity: A tool for rapid prototyping of physical interactive products. In Proceedings of DIS: ACM Conference on Designing Interactive Systems. pp. 141--46, 2002.

Digital Library

[7]

Ballagas, R., M. Ringel, M. Stone, and J. Borchers. iStuff: A physical user interface toolkit for ubiquitous computing environments. In Proceedings of CHI: ACM Conference on Human Factors in Computing Systems. pp. 537--44, 2003.

Digital Library

[8]

Barragan, H., Wiring: Prototyping Physical Interaction Design, Interaction Design Institute Ivrea, Italy, 2004.

[9]

Bellotti, V., M. Back, W. K. Edwards, R. E. Grinter, A. Henderson, and C. Lopes. Making sense of sensing systems: five questions for designers and researchers. In Proceedings of CHI: ACM Conference on Human Factors in Computing Systems. pp. 415--422, 2002.

Digital Library

[10]

Bevilacqua, F., R. Müller, and N. Schnell, MnM: a Max/MSP mapping toolbox. In Proceedings of NIME: Conference on New Interfaces for Musical Expression. pp. 85--88, 2005.

Digital Library

[11]

Blackwell, A. and T. Green, A Cognitive Dimensions Questionnaire, 2000. http://www.cl.cam.ac.uk/~afb21/CognitiveDimensions/CDquestionnaire.pdf

[12]

Card, S. K., J. D. Mackinlay, and G. G. Roebertson. A mor-phological analysis of the design space of input devices. ACM Trans. Inf. Systems. 9(2): ACM Press. pp. 99--122, 1991.

Digital Library

[13]

Cypher, A., ed. Watch What I Do -- Programming by Demonstration. MIT Press: Cambridge, MA. 652 pp., 1993.

Digital Library

[14]

Dey, A. K., R. Hamid, C. Beckmann, I. Li, and D. Hsu.a CAPpella: Programming by demonstration of context-aware applications. In Proceedings of CHI: ACM Conference on Human Factors in Computing Systems. pp 33--40, 2004.

Digital Library

[15]

Fails, J. and D. Olsen. A design tool for camera-based interaction. In Proceedings of CHI: ACM Conference on Human Factors in Computing Systems. pp 449--456, 2003.

Digital Library

[16]

Green, T. R. G. Cognitive dimensions of notations. People and Computers V. pp. 443--60, 1989.

Digital Library

[17]

Green, T. R. G. and M. Petre. Usability Analysis of Visual Programming Environments: A 'Cognitive Dimensions' Framework. Journal of Visual Languages and Computing 7(2). pp. 131--74, 1996.

[18]

Greenberg, S. and C. Fitchett. Phidgets: Easy development of physical interfaces through physical widgets. In Proceedings of UIST: ACM Symposium on User Interface Software and Technology. pp. 209--18, 2001.

Digital Library

[19]

Hartmann, B., S. R. Klemmer, M. Bernstein, L. Abdulla, B. Burr, A. Robinson-Mosher and J. Gee. Reflective physical prototyping through integrated design, test, and analysis. In Proceedings of UIST 2006: ACM Symposium on User Interface Software and Technology. pp. 299--308, 2006.

Digital Library

[20]

Hudson, S. and J. Mankoff. Rapid Construction of Functioning Physical Interfaces from Cardboard, Thumbtacks, Tin Foil and Masking Tape. In Proceedings of UIST: ACM Symposium on User Interface Software and Technology. pp 289--298, 2006.

Digital Library

[21]

Hutchins, E. L., J. D. Hollan, and D. A. Norman. Direct Manipulation Interfaces. Human-Computer Interaction 1(4). pp. 311--38, 1985.

Digital Library

[22]

Klemmer, S. R., B. Hartmann, and L. Takayama. How Bodies Matter: Five Themes for Interaction Design. In Proceedings of DIS: ACM Conference on Designing Interactive Systems. pp. 140--149, 2006.

Digital Library

[23]

Klemmer, S. R., J. Li, J. Lin, and J. A. Landay. Papier-Mâché: Toolkit Support for Tangible Input. In Proceedings of CHI: ACM Conference on Human Factors in Computing Systems. pp. 399--406, 2004.

Digital Library

[24]

Klemmer, S. R., A. K. Sinha, J. Chen, J. A. Landay, N. Aboo-baker, and A. Wang. SUEDE: A Wizard of Oz Prototyping Tool for Speech User Interfaces. In Proceedings of UIST: ACM Symposium on User Interface Software and Technology. pp. 1--10, 2000.

Digital Library

[25]

Li, Y. and J. A. Landay. Informal prototyping of continuous graphical interactions by demonstration. In Proceedings of UIST: ACM Symposium on User Interface Software and Technology. pp. 221--30, 2005.

Digital Library

[26]

Lieberman, H., ed. Your Wish is my Command. Morgan Kaufmann. 416 pp., 2001.

[27]

Lieberman, H., F. Paternò, and V. Wulf, ed. End-User Development. Springer: Dordrecht, Netherlands. 492 pp., 2006.

[28]

Merrill, D. J. and J. A. Paradiso, Personalization, Expressivity, and Learnability of an Implicit Mapping Strategy for Physical Interfaces. In Extended Abstracts of CHI: ACM Conference on Human Factors in Computing Systems. 2005.

[29]

Munch, S., J. Kreuziger, M. Kaiser, and R. Dillmann. Robot programming by demonstration -- using machine learning and user interaction methods for the development of easy and comfortable robot programming systems. In Proceedings of International Symposium on Industrial Robots. 1994.

[30]

Nardi, B. A., A Small Matter of Programming: Perspectives on End User Computing. Cambridge, MA: MIT Press. 1993.

[31]

Norman, D. A., Things that make us smart: Defending human attributes in the age of the machine: Addison Wesley Publishing Company. 290 pp., 1993.

Digital Library

[32]

Puckette, M., Pd, 2006. http://crca.ucsd.edu/~msp/software.html

[33]

Rogers, Y. and H. Muller. A framework for designing sensor-based interactions to promote exploration and reflection in play. International Journal of Human-Computer Studies 64(1). pp. --14, 2006.

Digital Library

[34]

Sakoe, H. and S. Chiba. Dynamic programming algorithm optimization for spoken word recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing 26(1): IEEE. pp. 43--49, 1978.

[35]

Selker, T., A bike helmet built for road hazards, 2006. http://news.com.com/2300-1008_3-6111157-2.html

[36]

Shneiderman, B., Overview + Detail, in Readings in Information Visualization, S.K. Card, J.D. Mackinlay, and B. Shneiderman, ed. Morgan Kaufmann, 1996.

[37]

Steiner, H.-C., {hid} toolkit: a unified framework for instrument design. In Proceedings of NIME: Conference on New Interfaces for Musical Expression. pp. 140--143, 2005.

Digital Library

[38]

Tufte, E. R., Envisioning Information. Cheshire, CT: Graphics Press LLC. 126 pp. 1990.

Digital Library

Cited By

Kianpisheh MMariakakis ATruong K(2024)exHARProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435008:1(1-30)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643500
Williams ABai MBuck JMckinney TRechkemmer AKalyanaraman KLease MHaffner PZhou XLi E(2024)Snapper: Accelerating Bounding Box Annotation in Object Detection Tasks with Find-and-Snap ToolingProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645162(471-488)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645162
Crisan AShang MBrochu E(2024)Eliciting Model Steering Interactions From Users via Data and Visual Design ProbesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332289830:9(6005-6019)Online publication date: Sep-2024
https://doi.org/10.1109/TVCG.2023.3322898
Show More Cited By

Index Terms

Authoring sensor-based interactions by demonstration with direct manipulation and pattern recognition
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Touch screens
    2. Interaction paradigms
2. Software and its engineering
  1. Software notations and tools
    1. Development frameworks and environments
      1. Integrated and visual development environments

Recommendations

An energy efficient Genetic Algorithm based approach for sensor-to-sink binding in multi-sink wireless sensor networks

Wireless sensor networks (WSNs) are ad-hoc networks in which sensors, that are designed to relay data back to sink nodes and/or Base Stations, are deployed in an area and may be configured in real time. Sensors, however, have limited energy supplies and ...
PaperPulse: An Integrated Approach to Fabricating Interactive Paper
CHI EA '15: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems

We present PaperPulse, a design and fabrication approach that enables designers to produce standalone interactive paper artifacts by augmenting them with electronics. With PaperPulse, users overlay visual designs with widgets provided in the design ...
A pattern for a sensor node
PLOP '10: Proceedings of the 17th Conference on Pattern Languages of Programs

Sensors are widely used in everyday life in household appliances, fire alarms, traffic control systems, battlefields, banks, and museums. Sensors are used either as standalone devices or in networks. Understanding the basic structure of a sensor node is ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '07: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

April 2007

1654 pages

ISBN:9781595935939

DOI:10.1145/1240624

General Chair:
Mary Beth Rosson
The Pennsylvania State University, USA
,
Program Chair:
David Gilmore
Intel, USA

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 April 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

CHI07

Sponsor:

CHI07: CHI Conference on Human Factors in Computing Systems

April 28 - May 3, 2007

California, San Jose, USA

Acceptance Rates

CHI '07 Paper Acceptance Rate 182 of 840 submissions, 22%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

107
Total Citations
View Citations
2,215
Total Downloads

Downloads (Last 12 months)52
Downloads (Last 6 weeks)12

Reflects downloads up to 02 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kianpisheh MMariakakis ATruong K(2024)exHARProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435008:1(1-30)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643500
Williams ABai MBuck JMckinney TRechkemmer AKalyanaraman KLease MHaffner PZhou XLi E(2024)Snapper: Accelerating Bounding Box Annotation in Object Detection Tasks with Find-and-Snap ToolingProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645162(471-488)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645162
Crisan AShang MBrochu E(2024)Eliciting Model Steering Interactions From Users via Data and Visual Design ProbesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332289830:9(6005-6019)Online publication date: Sep-2024
https://doi.org/10.1109/TVCG.2023.3322898
Sayara AChen ENguyen CXiao RYoon D(2023)GestureCanvas: A Programming by Demonstration System for Prototyping Compound Freehand Interaction in VRProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606736(1-17)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606736
Chauvergne EHachet MProuzeau A(2023)Authoring Interactive and Immersive Experiences Using Programming by DemonstrationProceedings of the 34th Conference on l'Interaction Humain-Machine10.1145/3583961.3583981(1-13)Online publication date: 3-Apr-2023
https://dl.acm.org/doi/10.1145/3583961.3583981
Chong TShin HAneja DIgarashi T(2023)SoundToons: Exemplar-Based Authoring of Interactive Audio-Driven Animation SpritesProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584047(710-722)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584047
Nicholas MMarquardt NPahud MRiche NRomat HCollins CLedo DKadekodi RChandramouli BHinckley K(2023)Escapement: A Tool for Interactive Prototyping with Video via Sensor-Mediated Abstraction of TimeProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581115(1-14)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581115
Ye HLeng JXiao CWang LFu H(2023)ProObjAR: Prototyping Spatially-aware Interactions of Smart Objects with AR-HMDProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580750(1-15)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580750
He FHu XShi JQian XWang TRamani K(2023)Ubi Edge: Authoring Edge-Based Opportunistic Tangible User Interfaces in Augmented RealityProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580704(1-14)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580704
Azim MRahman AHeo S(2023)SequenceSenseInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2023.103035176:COnline publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1016/j.ijhcs.2023.103035
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents