research-article

Real-Time Screen-Camera Communication Behind Any Scene

Authors:

Andrew T. Campbell,

Xia ZhouAuthors Info & Claims

MobiSys '15: Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services

Pages 197 - 211

https://doi.org/10.1145/2742647.2742667

Published: 18 May 2015 Publication History

Abstract

We present HiLight, a new form of real-time screen-camera communication without showing any coded images (e.g., barcodes) for off-the-shelf smart devices. HiLight encodes data into pixel translucency change atop any screen content, so that camera-equipped devices can fetch the data by turning their cameras to the screen. HiLight leverages the alpha channel, a well-known concept in computer graphics, to encode bits into the pixel translucency change. By removing the need to directly modify pixel RGB values, HiLight overcomes the key bottleneck of existing designs and enables real-time unobtrusive communication while supporting any screen content. We build a HiLight prototype using off-the-shelf smart devices and demonstrate its efficacy and robustness in practical settings. By offering an unobtrusive, flexible, and lightweight communication channel between screens and cameras, HiLight opens up opportunities for new HCI and context-aware applications, e.g., smart glasses communicating with screens to realize augmented reality.

References

[1]

https://www.opengl.org/.

[2]

http://opencv.org/platforms/android.html.

[3]

http://developer.android.com/guide/topics/resources/drawable-resource.html.

[4]

https://developer.android.com/about/versions/android-5.0.html.

[5]

http://en.wikipedia.org/wiki/OLED.

[6]

Automatic identification and data capture techniques - QR code 2005 bar code symbology specification. ISO/IEC 18004:2006.

[7]

ASHOK, A., ET AL. Challenge: Mobile optical networks through visual MIMO. In Proc. of MobiCom (2010).

Digital Library

[8]

BERGER, M. Geometry I. Springer Science & Business, 2009.

[9]

BULLOUGH, J., ET AL. Effects of flicker characteristics from solid-state lighting on detection, acceptability and comfort. Lighting Research and Technology 43, 3 (2011), 337--348.

[10]

CARVALHO, R., CHU, C.-H., AND CHEN, L.-J. IVC: Imperceptible video communication. In Proc. of HotMobile (poster) (2014).

[11]

DAVEY, M. C., AND MACKAY, D. J. Low density parity check codes over gf (q). In Information Theory Workshop, 1998 (1998), IEEE, pp. 70--71.

[12]

DELL, N., VAIDYANATHAN, V., MEDHI, I., CUTRELL, E., AND THIES, W. "Yours is Better!": Participant Response Bias in HCI. In Proc. of CHI (2012).

Digital Library

[13]

FATTAL, D., ET AL. A multi-directional backlight for a wide-angle, glasses-free three-dimensional display. Nature 495, 7441 (2013), 348--351.

[14]

HAO, T., ZHOU, R., AND XING, G. COBRA: Color barcode streaming for smartphone systems. In Proc. of MobiSys (2012).

Digital Library

[15]

HU, W., GU, H., AND PU, Q. LightSync: Unsynchronized visual communication over screen-camera links. In Proc. of MobiCom (2013).

Digital Library

[16]

HU, W., MAO, J., HUANG, Z., XUE, Y., SHE, J., BIAN, K., AND SHEN, G. Strata: Layered coding for scalable visual communication. In Proc. of MobiCom (2014).

Digital Library

[17]

HUANG, C.-L., AND LIAO, B.-Y. A robust scene-change detection method for video segmentation. IEEE Transactions on Circuits and Systems for Video Technology 11, 12 (2001), 1281--1288.

Digital Library

[18]

HUANG, S. Backlight modulation circuit having rough and fine illumination signal processing circuit, Mar. 27 2012. US Patent 8,144,112.

[19]

HUANG, W., AND MOW, W. H. PiCode: 2D barcode with embedded picture and ViCode: 3D barcode with embedded video (poster). In Proc. of MobiCom (2013).

Digital Library

[20]

KIMURA, K., MASUDA, S., AND HAYASHI, M. Display apparatus and method for controlling a backlight with multiple light sources of a display unit, Sept. 11 2012. US Patent 8,264,447.

[21]

LI, T., AN, C., CAMPBELL, A., AND ZHOU, X. HiLight: Hiding bits in pixel translucency changes. In Proc. of the 1st ACM MobiCom Workshop on Visible Light Communication Systems (VLCS) (2014).

Digital Library

[22]

LIKAMWA, R., RAMIREZ, D., AND HOLLOWAY, J. Styrofoam: A tightly packed coding scheme for camera-based visible light communication. In Proc. of the 1st ACM MobiCom Workshop on Visible Light Communication Systems (VLCS) (2014).

Digital Library

[23]

MOHAN, A., WOO, G., HIURA, S., SMITHWICK, Q., AND RASKAR, R. Bokode: Imperceptible visual tags for camera based interaction from a distance. In Proc. of SIGGRAPH (2009).

Digital Library

[24]

MUNSHI, A., GINSBURG, D., AND SHREINER, D. OpenGL ES 2.0 programming guide. Pearson Education, 2008.

Digital Library

[25]

NAGASAKA, A., AND TANAKA, Y. Automatic video indexing and full-video search for object appearances. In Proc. of the IFIP TC2/WG 2.6 Second Working Conference on Visual Database Systems II (1992).

Digital Library

[26]

PERLI, S. D., AHMED, N., AND KATABI, D. PixNet: Interference-free wireless links using LCD-camera pairs. In Proc. of MobiCom (2010).

Digital Library

[27]

PORTER, T., AND DUFF, T. Compositing digital images. In ACM SIGGRAPH Computer Graphics (1984).

Digital Library

[28]

RAJAGOPAL, N., LAZIK, P., AND ROWE, A. Visual light landmarks for mobile devices. In Proc. of IPSN (2014).

Digital Library

[29]

ROSIN, P. L. Thresholding for change detection. In Computer Vision, 1998. Sixth International Conference on (1998), IEEE, pp. 274--279.

Digital Library

[30]

ROSIN, P. L., AND IOANNIDIS, E. Evaluation of global image thresholding for change detection. Pattern Recognition Letters 24, 14 (2003), 2345--2356.

Digital Library

[31]

SETHI, I. K., AND PATEL, N. V. Statistical approach to scene change detection. In IS&T/SPIE?s Symposium on Electronic Imaging: Science & Technology (1995), pp. 329--338.

[32]

SMITS, P. C., AND ANNONI, A. Toward specification-driven change detection. IEEE Transactions on Geoscience and Remote Sensing 38, 3 (2000), 1484--1488.

[33]

SUR, S., WEI, T., AND ZHANG, X. Autodirective audio capturing through a synchronized smartphone array. In ACM Mobisys 2014.

Digital Library

[34]

TAN, K. W., ET AL. FOCUS: a usable & effective approach to OLED display power management. In UbiComp (2013).

Digital Library

[35]

VAN DE SANDE, J., AND ASSIGNOR, T. Real-time beamforming and sound classification parameter generation in public environments. TNO report TNO-DV (2012), S007.

[36]

WANG, A., MA, S., HU, C., HUAI, J., PENG, C., AND SHEN, G. Enhancing reliability to boost the throughput over screen-camera links. In Proc. of MobiCom (2014).

Digital Library

[37]

WANG, A., PENG, C., ZHANG, O., SHEN, G., AND ZENG, B. InFrame: Multiflexing full-frame visible communication channel for humans and devices. In Proc. of HotNets (2014).

Digital Library

[38]

WICKER, S. B., AND BHARGAVA, V. K. Reed-Solomon codes and their applications. John Wiley & Sons, 1999.

[39]

WOO, G., LIPPMAN, A., AND RASKAR, R. VRCodes: Unobtrusive and active visual codes for interaction by exploiting rolling shutter. In Proc. of ISMAR (2012).

Digital Library

[40]

YONEZAWA, T., OGAWA, M., KYONO, Y., NOZAKI, H., NAKAZAWA, J., NAKAMURA, O., AND TOKUDA, H. SENSeTREAM: Enhancing online live experience with sensor-federated video stream using animated two-dimensional code. In Proc. of UbiComp (2014).

Digital Library

[41]

YUAN, W., DANA, K., VARGA, M., ASHOK, A., GRUTESER, M., AND MANDAYAM, N. Computer vision methods for visual mimo optical systems. Proceedings of the IEEE International Workshop on Projector-Camera Systems (held with CVPR) (2011), 37--43.

[42]

YUAN, W., ET AL. Dynamic and invisible messaging for visual mimo. In IEEE Workshop on Applications of Computer Vision (WACV) (2012).

Digital Library

[43]

ZHANG, B., ET AL. SBVLC: Secure barcode-based visible light communication for smartphones. In Proc. of INFOCOM (2014).

[44]

ZHANG, H., KANKANHALLI, A., AND SMOLIAR, S. W. Automatic partitioning of full-motion video. Multimedia Systems 1, 1 (1993), 10--28.

Digital Library

Cited By

Su HGao LLiu TKneip L(2025)Motion-Aware Optical Camera Communication With Event CamerasIEEE Robotics and Automation Letters10.1109/LRA.2024.351729210:2(1385-1392)Online publication date: Mar-2025
https://doi.org/10.1109/LRA.2024.3517292
Singh PKim YKim BJung S(2024)Display Field Communication: Enabling Seamless Data Exchange in Screen–Camera EnvironmentsPhotonics10.3390/photonics1111100011:11(1000)Online publication date: 24-Oct-2024
https://doi.org/10.3390/photonics11111000
Wang YShen YXu KHassan MZhao GXu CHu WShu YLiu JTan RHe YChen J(2024)Towards High-Speed Passive Visible Light Communication with Event Cameras and Digital Micro-MirrorsProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699368(704-717)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3666025.3699368
Show More Cited By

Index Terms

Real-Time Screen-Camera Communication Behind Any Scene
1. Networks
  1. Network types
    1. Mobile networks
    2. Wireless access networks

Recommendations

Screen-Camera Communication via Matrix Barcode Utilizing Imperceptible Color Vibration
UIST '18 Adjunct: Adjunct Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology

Communication between screens and cameras has attracted attention as a ubiquitous information source, motivated by the widespread use of smartphones and the increase of public advertising and information screens. We propose embedding matrix barcodes ...
Deep screen space
I3D '14: Proceedings of the 18th meeting of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games

Computing shading such as ambient occlusion (AO), subsurface scattering (SSS) or indirect light (GI) in screen space has recently received a lot of attention. While being efficient to compute, screen space methods have several key limitations such as ...
Capacity of screen-camera communications under perspective distortions

Cameras are ubiquitous and increasingly being used not just for capturing images but also for communicating information. For example, the pervasive QR codes can be viewed as communicating a short code to camera-equipped sensors. Such communications ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MobiSys '15: Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services

May 2015

516 pages

ISBN:9781450334945

DOI:10.1145/2742647

General Chairs:
Gaetano Borriello
University of Washington, USA
,
Giovanni Pau
UPMC-LIP6, France / UCLA, USA
,
Program Chairs:
Marco Gruteser
Rutgers University, USA
,
Jason Hong
Carnegie Mellon University, USA

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMOBILE: ACM Special Interest Group on Mobility of Systems, Users, Data and Computing

In-Cooperation

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 May 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation
Dartmouth College

Conference

MobiSys'15

Sponsor:

SIGMOBILE

MobiSys'15: The 13th Annual International Conference on Mobile Systems, Applications, and Services

May 18 - 22, 2015

Florence, Italy

Acceptance Rates

MobiSys '15 Paper Acceptance Rate 29 of 219 submissions, 13%;

Overall Acceptance Rate 274 of 1,679 submissions, 16%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

92
Total Citations
View Citations
913
Total Downloads

Downloads (Last 12 months)77
Downloads (Last 6 weeks)5

Reflects downloads up to 28 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Su HGao LLiu TKneip L(2025)Motion-Aware Optical Camera Communication With Event CamerasIEEE Robotics and Automation Letters10.1109/LRA.2024.351729210:2(1385-1392)Online publication date: Mar-2025
https://doi.org/10.1109/LRA.2024.3517292
Singh PKim YKim BJung S(2024)Display Field Communication: Enabling Seamless Data Exchange in Screen–Camera EnvironmentsPhotonics10.3390/photonics1111100011:11(1000)Online publication date: 24-Oct-2024
https://doi.org/10.3390/photonics11111000
Wang YShen YXu KHassan MZhao GXu CHu WShu YLiu JTan RHe YChen J(2024)Towards High-Speed Passive Visible Light Communication with Event Cameras and Digital Micro-MirrorsProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699368(704-717)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3666025.3699368
Xiao RZhao LQian FYang LHan JOkoshi TKo JLiKamWa R(2024)Practical Optical Camera Communication Behind Unseen and Complex BackgroundsProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661866(113-126)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3643832.3661866
Dang FXu YXu RChen XLiu Y(2024)LSync: A Universal Timeline-Synchronizing Solution for Live StreamingIEEE/ACM Transactions on Networking10.1109/TNET.2024.340814732:5(4144-4159)Online publication date: Oct-2024
https://doi.org/10.1109/TNET.2024.3408147
Liao ZLuo ZHuang QZhang LWu FZhang QChen G(2024)Gesture Recognition Using Visible Light on Mobile DevicesIEEE/ACM Transactions on Networking10.1109/TNET.2024.336999632:4(2920-2935)Online publication date: Aug-2024
https://doi.org/10.1109/TNET.2024.3369996
Li YLiao XWu X(2024)Screen-Shooting Resistant Watermarking With Grayscale Deviation SimulationIEEE Transactions on Multimedia10.1109/TMM.2024.341541526(10908-10923)Online publication date: 2024
https://doi.org/10.1109/TMM.2024.3415415
Han HXie KWang TZhu XZhao YXu F(2024)RescQR: Enabling Reliable Data Recovery in Screen-Camera Communication SystemIEEE Transactions on Mobile Computing10.1109/TMC.2023.3277212(1-13)Online publication date: 2024
https://doi.org/10.1109/TMC.2023.3277212
Zhao ZQiu YZou GWeng JYang BQin Z(2024)18‐3: Mini‐LED LCDs Integrated with High‐capacity MIMO Visible Light CommunicationSID Symposium Digest of Technical Papers10.1002/sdtp.1749255:1(212-215)Online publication date: 30-Jul-2024
https://doi.org/10.1002/sdtp.17492
Mazurek PBak D(2023)Video Sequence Segmentation Based on K-Means in Air-Gap Data Transmission for a Cluttered EnvironmentSensors10.3390/s2302066523:2(665)Online publication date: 6-Jan-2023
https://doi.org/10.3390/s23020665
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

EPUB

View this article in ePub.

Figures

Tables

Media

View Table of Conten