research-article

Combining multiple depth cameras and projectors for interactions on, above and between surfaces

Authors:

Andrew D. Wilson,

Hrvoje BenkoAuthors Info & Claims

UIST '10: Proceedings of the 23nd annual ACM symposium on User interface software and technology

Pages 273 - 282

https://doi.org/10.1145/1866029.1866073

Published: 03 October 2010 Publication History

Abstract

Instrumented with multiple depth cameras and projectors, LightSpace is a small room installation designed to explore a variety of interactions and computational strategies related to interactive displays and the space that they inhabit. LightSpace cameras and projectors are calibrated to 3D real world coordinates, allowing for projection of graphics correctly onto any surface visible by both camera and projector. Selective projection of the depth camera data enables emulation of interactive displays on un-instrumented surfaces (such as a standard table or office desk), as well as facilitates mid-air interactions between and around these displays. For example, after performing multi-touch interactions on a virtual object on the tabletop, the user may transfer the object to another display by simultaneously touching the object and the destination display. Or the user may "pick up" the object by sweeping it into their hand, see it sitting in their hand as they walk over to an interactive wall display, and "drop" the object onto the wall by touching it with their other hand. We detail the interactions and algorithms unique to LightSpace, discuss some initial observations of use and suggest future directions.

Supplementary Material

JPG File (293s-wilson.jpg)

Download
4.43 KB

MP4 File (293s-wilson.mp4)

Download
42.47 MB

References

[1]

}}Bandyopadhyay, D., Raskar, R., and Fuchs, H. (2001). Dynamic shader lamps: Painting on movable objects. In Proc. of IEEE and ACM International Symposium on Augmented Reality (ISAR '01). 207--216.

Digital Library

[2]

}}Benko, H., and Wilson, A. (2009). DepthTouch: Using Depth-Sensing Camera to Enable Freehand Interactions On and Above the Interactive Surface. Microsoft Research Technical Report MSR-TR-2009-23.

[3]

}}Brooks, R. A., Coen, M., Dang, D., Bonet, J. D., Kramer, J., Lozano-Perez, T., Mellor, J., Pook, P., Stauffer, C., Stein, L., Torrance, M. and Wessler, M. (1997). The Intelligent Room Project. In Proc. of International Conference on Cognitive Technology (CT '97). 271--278.

Digital Library

[4]

}}Cao, X., Forlines, C., and Balakrishnan, R. (2007). Multi-user interaction using handheld projectors. In Proc. of ACM UIST '07. 43--52.

Digital Library

[5]

}}Cruz-Neira, C., Sandin, D. J., and DeFanti, T. A. (1993). Surround-screen projection-based virtual reality: The design and implementation of the CAVE. In Proc. of ACM SIGGRAPH 93. 135--142.

Digital Library

[6]

}}DeMenthon D. and Davis, L. S. (1995). Model-Based Object Pose in 25 Lines of Code. International Journal of Computer Vision, vol. 15, June 1995. 123--141.

Digital Library

[7]

}}Deutscher, J. and Reid, I. (2005). Articulated Body Motion Capture by Stochastic Search. Int. Journal of Computer Vision 61, 2 (Feb.). 185--205.

Digital Library

[8]

}}Fails, J. A., and Olsen, D. R. (2002) LightWidgets: Interacting in Everyday Spaces. In Proc. of IUI '02. 63--69.

Digital Library

[9]

}}Harrison, C., Tan, D., and Morris, D. (2010). Skinput: Appropriating the Body as an Input Surface. In Proc. of ACM SIGCHI '10. 453--462.

Digital Library

[10]

}}Hilliges, O., Izadi, S., Wilson, A. D., Hodges, S., Garcia-Mendoza, A., and Butz, A. (2009). Interactions in the Air: Adding Further Depth to Interactive Tabletops. In Proc. of ACM UIST '09. 139--148.

Digital Library

[11]

}}Hinckley, K., Pausch, R., Goble, J. C., and Kassell, N. F. (1994). A Survey of Design Issues in Spatial Input. In Proc. of ACM UIST '94. 213--222.

Digital Library

[12]

}}Holman, D. and Vertegaal, R. (2008). Organic user interfaces: designing computers in any way, shape, or form. Comm. of the ACM 51, 6 (Jun. 2008). 48--55.

Digital Library

[13]

}}Horn, B. K. P. (1987). Closed Form Solution of Absolute Orientation Using Unit Quaternions. J. Opt. Soc. Am. A, 4, 629--642.

[14]

}}Hua, H., Brown, L. D., and Gao, C. (2004). Scape: Supporting Stereoscopic Collaboration in Augmented and Projective Environments. IEEE Comput. Graph. Appl. 24, 1 (Jan. 2004). 66--75.

Digital Library

[15]

}}Johanson, B., Fox, A. and Winograd, T. (2002). The Interactive Workspaces Project: Experiences with Ubiquitous Computing Rooms. IEEE Pervasive Computing, Vol. 1 (2). 67--74.

Digital Library

[16]

}}Krumm, J., Harris, S., Meyers, B., Brumitt, B., Hale, M., and Shafer, S. (2000). Multi-camera multi-person tracking for EasyLiving. In Proc. of IEEE International Workshop on Visual Surveillance. 3--10.

Digital Library

[17]

}}Lee, J. C., Hudson, S. E., Summet, J. W., and Dietz, P. H. (2005). Moveable interactive projected displays using projector based tracking. In Proc. of ACM UIST '05. 63--72.

Digital Library

[18]

}}Mistry, P., and Maes, P. (2009) SixthSense - A Wearable Gestural Interface. SIGGRAPH Asia '09, Emerging Technologies. Yokohama, Japan.

Digital Library

[19]

}}Pinhanez, C. S. (2001). The Everywhere Displays Projector: A Device to Create Ubiquitous Graphical Interfaces. In Proc. of the International Conference on Ubiquitous Computing (UBICOMP). 315--331.

Digital Library

[20]

}}Piper, B., Ratti, C., and Ishii, H. (2002) Illuminating Clay: A 3-D Tangible Interface for Landscape Analysis. In Proc. of ACM SIGCHI '02. 355--362.

Digital Library

[21]

}}Raskar, R., Welch, G., Cutts, M., Lake, A., Stesin, L., and Fuchs, H. (1998). The Office of the Future: A Unified Approach to Image-Based Modeling and Spatially Immersive Displays. In Proc. of ACM SIGGRAPH '98. 179--188.

Digital Library

[22]

}}Rekimoto, J. and Saitoh, M. (1999). Augmented Surfaces: A Spatially Continuous Work Space for Hybrid Computing Environments. In Proc. of ACM SIGCHI '99. 378--385.

Digital Library

[23]

}}Starner, T., Leibe, B., Minnen, D., Westeyn, T., Hurst, A., and Weeks, J. (2003). The Perceptive Workbench: Computer-Vision-Based Gesture Tracking, Object Tracking, and 3D Reconstruction for Augmented Desks. Machine Vision and Applications, vol. 14, 51--71.

Digital Library

[24]

}}Streitz, N., Geißler, J., Holmer, T., Konomi, S., Müller-Tomfelde, C., Reischl, W., Rexroth, P., Seitz, P., and Steinmetz, R. (1999). i-LAND: An Interactive Landscape for Creativity and Innovation. In Proc. of ACM SIGCHI '99. 120--127.

Digital Library

[25]

}}Underkoffler, J., Ullmer, B., and Ishii, H. (1999). Emancipated pixels: Real-world graphics in the luminous room. In Proc. of ACM SIGGRAPH '99. 385--392.

Digital Library

[26]

}}Wellner, P. (1993). Interacting with paper on the DigitalDesk. Communications of the ACM. 36, 7 (Jul. 1993). 87--96.

Digital Library

[27]

}}Wilson, A. (2005). PlayAnywhere: A Compact Tabletop Computer Vision System. In Proc. of ACM UIST '05. 83--92.

Digital Library

[28]

}}Wilson, A. (2007) Depth-Sensing Video Cameras for 3D Tangible Tabletop Interaction. In Proc. of IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP '07). 201--204.

[29]

}}Wilson, A. D., Izadi, S., Hilliges, O., Garcia-Mendoza, A., and Kirk, D. (2008). Bringing physics to the surface. In Proc. of ACM UIST '08. 67--76.

Digital Library

[30]

}}Wren, C., Azarbayejani, A., Darrell, T., and Pentland, A. (1997). Pfinder: real-time tracking of the human body, IEEE Trans. PAMI 19 (7).

Digital Library

[31]

}}Yilmaz, A., Javed, O., and Shah, M. (2006). Object tracking: A survey. ACM Comput. Surv. 38, 4 (Dec. '06), Article #13.

Digital Library

Cited By

Böhmer MKuehnel SDamarowsky J(2024)Bridging the Gap Between Augmentation and CognitionImpact of Digital Solutions for Improved Healthcare Delivery10.4018/979-8-3693-5237-3.ch016(409-444)Online publication date: 18-Oct-2024
https://doi.org/10.4018/979-8-3693-5237-3.ch016
Vanderdonckt JVatavu RSluÿters A(2024)Engineering Touchscreen Input for 3-Way Displays: Taxonomy, Datasets, and ClassificationCompanion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems10.1145/3660515.3661331(57-65)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3660515.3661331
Daeijavad P(2024)Investigating the Impact of Multiple View Layouts on Users' Visual Task Performance in Extended RealityProceedings of the 2024 International Conference on Advanced Visual Interfaces10.1145/3656650.3656756(1-3)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3656650.3656756
Show More Cited By

Index Terms

Combining multiple depth cameras and projectors for interactions on, above and between surfaces
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Graphical user interfaces

Recommendations

A portable system for anywhere interactions
CHI EA '04: CHI '04 Extended Abstracts on Human Factors in Computing Systems

Interactions have taken off from the confinement of a single screen into various personal devices. Projected an interface onto different parts of a physical environment is an escape beyond traditional display devices. Imagine that any walls or floors ...
Design and evaluation of proxemics-aware environments to support navigation in large information spaces
CHI EA '13: CHI '13 Extended Abstracts on Human Factors in Computing Systems

In my research, I explore the use of proxemics in Human-Computer Interaction to design explicit and implicit interaction with knowledge work environments for literature review, reading & writing, or discussion. This paper proposes the employment of ...
Steerable augmented reality with the beamatron
UIST '12: Proceedings of the 25th annual ACM symposium on User interface software and technology

Steerable displays use a motorized platform to orient a projector to display graphics at any point in the room. Often a camera is included to recognize markers and other objects, as well as user gestures in the display volume. Such systems can be used ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '10: Proceedings of the 23nd annual ACM symposium on User interface software and technology

October 2010

476 pages

ISBN:9781450302715

DOI:10.1145/1866029

General Chair:
Ken Perlin
New York University
,
Program Chairs:
Mary Czerwinski
Microsoft Research
,
Rob Miller
MIT CSAIL

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 October 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

UIST '10

Sponsor:

UIST '10: The 23nd Annual ACM Symposium on User Interface Software and Technology

October 3 - 6, 2010

New York, New York, USA

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

245
Total Citations
View Citations
4,515
Total Downloads

Downloads (Last 12 months)75
Downloads (Last 6 weeks)6

Reflects downloads up to 11 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Böhmer MKuehnel SDamarowsky J(2024)Bridging the Gap Between Augmentation and CognitionImpact of Digital Solutions for Improved Healthcare Delivery10.4018/979-8-3693-5237-3.ch016(409-444)Online publication date: 18-Oct-2024
https://doi.org/10.4018/979-8-3693-5237-3.ch016
Vanderdonckt JVatavu RSluÿters A(2024)Engineering Touchscreen Input for 3-Way Displays: Taxonomy, Datasets, and ClassificationCompanion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems10.1145/3660515.3661331(57-65)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3660515.3661331
Daeijavad P(2024)Investigating the Impact of Multiple View Layouts on Users' Visual Task Performance in Extended RealityProceedings of the 2024 International Conference on Advanced Visual Interfaces10.1145/3656650.3656756(1-3)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3656650.3656756
Streli PRichardson MBotros FMa SWang RHolz C(2024)TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric VisionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676330(1-16)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676330
Cao YKazi RWei LAneja DXia H(2024)Elastica: Adaptive Live Augmented Presentations with Elastic Mappings Across ModalitiesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642725(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642725
Dupré CAppert CRey SSaidi HPietriga E(2024)TriPad: Touch Input in AR on Ordinary Surfaces with Hand Tracking OnlyProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642323(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642323
Wang XSu ZRekimoto JZhang Y(2024)Watch Your Mouth: Silent Speech Recognition with Depth SensingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642092(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642092
Mendoza-Ramírez CTudon-Martinez JFélix-Herrán LLozoya-Santos JVargas-Martínez A(2023)Augmented Reality: SurveyApplied Sciences10.3390/app13181049113:18(10491)Online publication date: 20-Sep-2023
https://doi.org/10.3390/app131810491
Streli PJiang JRossie JHolz C(2023)Structured Light Speckle: Joint Ego-Centric Depth Estimation and Low-Latency Contact Detection via Remote VibrometryProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606749(1-12)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606749
Catană AVatavu R(2023)Fingerhints: Understanding Users’ Perceptions of and Preferences for On-Finger Kinesthetic NotificationsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581022(1-17)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581022
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents