research-article

iSpy: automatic reconstruction of typed input from compromising reflections

Authors:

Andrew M. White,

Dibyendusekhar Goswami,

Fabian Monrose,

Jan-Michael FrahmAuthors Info & Claims

CCS '11: Proceedings of the 18th ACM conference on Computer and communications security

Pages 527 - 536

https://doi.org/10.1145/2046707.2046769

Published: 17 October 2011 Publication History

Abstract

We investigate the implications of the ubiquity of personal mobile devices and reveal new techniques for compromising the privacy of users typing on virtual keyboards. Specifi- cally, we show that so-called compromising reflections (in, for example, a victim's sunglasses) of a device's screen are sufficient to enable automated reconstruction, from video, of text typed on a virtual keyboard. Despite our deliberate use of low cost commodity video cameras, we are able to compensate for variables such as arbitrary camera and device positioning and motion through the application of advanced computer vision and machine learning techniques. Using footage captured in realistic environments (e.g., on a bus), we show that we are able to reconstruct fluent translations of recorded data in almost all of the test cases, correcting users' typing mistakes at the same time. We believe these results highlight the importance of adjusting privacy expectations in response to emerging technologies.

References

[1]

D. Asonov and R. Agrawal. Keyboard acoustic emanations. In Proceedings of IEEE Symposium on Security and Privacy, 2004.

[2]

uth, and Unruh}Backes:2008M. Backes, M. Dürmuth, and D. Unruh. Compromising reflections-or-how to read LCD monitors around the corner. In Proceedings of the IEEE Symposium on Security and Privacy, 2008.

Digital Library

[3]

M. Backes, T. Chen, M. Duermuth, H. Lensch, and M. Welk. Tempest in a teapot: Compromising reflections revisited. In Proceedings of the IEEE Symposium on Security and Privacy, 2009.

Digital Library

[4]

D. Balzarotti, M. Cova, and G. Vigna. ClearShot: Eavesdropping on keyboard input from video. In Proceedings of the IEEE Symposium on Security and Privacy, 2008.

Digital Library

[5]

M. Denkowski and A. Lavie. Choosing the right evaluation for machine translation: an examination of annotator and automatic metric performance on human judgment tasks. In Proceedings of the AMTA, 2010.

[6]

W. N. Francis and H. Kucera. Brown corpus manual. Technical report, Dept. of Linguistics, Brown University, 1979.

[7]

Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. In Proceedings of the 2nd European Conf. on Computational Learning Theory, pages 23--37, 1995.

Digital Library

[8]

H. Grabner, M. Grabner, and H. Bischof. Real-time tracking via on-line boosting. In British Machine Vision Conference, volume 1, pages 47--56, 2006.

[9]

H. Grabner, C. Leistner, and H. Bischof. Semi-supervised on-line boosting for robust tracking. European Conf. on Computer Vision, pages 234--247, 2008.

Digital Library

[10]

R. I. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, 2000.

Digital Library

[11]

H. J. Highland. Electromagnetic radiation revisited. Computer Security, 5: 85--93, June 1986.

Digital Library

[12]

P. J. Huber. Robust Statistics. John Wiley & Sons, 1981.

[13]

K. Jung, K. I. Kim, and A. K. Jain. Text information extraction in images and video: a survey. Pattern Recognition, 37 (5): 977 -- 997, 2004.

[14]

D. Jurafsky and J. H. Martin. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Prentice Hall, 2008.

Digital Library

[15]

Z. Kalal, K. Mikolajczyk, and J. Matas. Forward-backward error: Automatic detection of tracking failures. Int. Conference on Pattern Recognition, 2010.

Digital Library

[16]

M. Kuhn. Electromagnetic eavesdropping risks of flat-panel displays. In Privacy Enhancing Technologies, 2004.

Digital Library

[17]

M. G. Kuhn. Optical time-domain eavesdropping risks of CRT displays. In Proceedings of the IEEE Symposium on Security and Privacy, 2002.

Digital Library

[18]

A. Lavie. Evaluating the output of machine translation systems. AMTA Tutorial, 2010.

[19]

A. Lavie and M. J. Denkowski. The METEOR metric for automatic evaluation of machine translation. Machine Translation, 23: 105--115, September 2009.

Digital Library

[20]

K. Levi and Y. Weiss. Learning object detection from a small number of examples: the importance of good features. In Computer Vision and Pattern Recognition, 2004.

Digital Library

[21]

J. Loughry and D. A. Umphress. Information leakage from optical emanations. ACM TISSEC, 5: 262--289, August 2002.

Digital Library

[22]

D. Lowe. Distinctive image features from scale-invariant keypoints. Int. Journal of Computer Vision, 60 (2): 91--110, 2004.

Digital Library

[23]

NSA. TEMPEST: A signal problem. Cryptologic Spectrum, 2 (3), 1972.

[24]

en, and Maenpaa}Ojala2002T. Ojala, M. Pietikainen, and T. Maenpaa. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Machine Intelligence, 24: 971--987, July 2002.

Digital Library

[25]

F. C. N. Pereira and M. Riley. Speech recognition by composition of weighted finite automata. The Computing Research Repository, cmp-lg/9603001, 1996.

[26]

R. Raguram, J.-M. Frahm, and M. Pollefeys. A comparative analysis of RANSAC techniques leading to adaptive real-time random sample consensus. In European Conference on Computer Vision, pages II: 500--513, 2008.

Digital Library

[27]

S. Stalder, H. Grabner, and L. V. Gool. Beyond semi-supervised tracking: Tracking should be as simple as detection, but not simpler than recognition. Workshop on On-line Learning for Computer Vision, pages 1409--1416, Sept. 2009.

[28]

W. van Eck. Electromagnetic radiation from video display units: an eavesdropping risk? Computer Security, 4: 269--286, December 1985.

Digital Library

[29]

P. A. Viola and M. J. Jones. Rapid object detection using a boosted cascade of simple features. In Computer Vision and Pattern Recognition, 2001.

[30]

P. A. Viola and M. J. Jones. Robust real-time face detection. Int. Journal of Computer Vision, 57 (2): 137--154, 2004.

Digital Library

[31]

M. Vuagnoux and S. Pasini. Compromising electromagnetic emanations of wired and wireless keyboards. In Proceedings of the $18^th$ USENIX Security Symposium, pages 1--16, 2009.

Digital Library

[32]

L. Zhuang, F. Zhou, and J. D. Tygar. Keyboard acoustic emanations revisited. ACM TISSEC, 13, November 2009.

Digital Library

Cited By

Nassi BIluz ECohen OVayner ONassi DZadov BElovici Y(2024)Video-Based Cryptanalysis: Extracting Cryptographic Keys from Video Footage of a Device’s Power LED Captured by Standard Video Cameras2024 IEEE Symposium on Security and Privacy (SP)10.1109/SP54263.2024.00163(2422-2440)Online publication date: 19-May-2024
https://doi.org/10.1109/SP54263.2024.00163
Qiao HWang KHuang TXu XLiu SChen J(2024)A TEMPEST Attack Implementation based on Hidden Markov model in Smart GridJournal of Physics: Conference Series10.1088/1742-6596/2774/1/0120092774:1(012009)Online publication date: 1-Jul-2024
https://doi.org/10.1088/1742-6596/2774/1/012009
Farzand HAbraham MBrewster SKhamis MMarky K(2024)A Systematic Deconstruction of Human-Centric Privacy & Security Threats on Mobile PhonesInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2361519(1-24)Online publication date: 12-Jun-2024
https://doi.org/10.1080/10447318.2024.2361519
Show More Cited By

Index Terms

iSpy: automatic reconstruction of typed input from compromising reflections
1. Security and privacy
  1. Human and societal aspects of security and privacy
2. Social and professional topics
  1. Computing / technology policy
    1. Privacy policies

Recommendations

Seeing double: reconstructing obscured typed input from repeated compromising reflections
CCS '13: Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security

Of late, threats enabled by the ubiquitous use of mobile devices have drawn much interest from the research community. However, prior threats all suffer from a similar, and profound, weakness - namely the requirement that the adversary is either within ...
On the Privacy Risks of Virtual Keyboards: Automatic Reconstruction of Typed Input from Compromising Reflections

We investigate the implications of the ubiquity of personal mobile devices and reveal new techniques for compromising the privacy of users typing on virtual keyboards. Specifically, we show that so-called compromising reflections (in, for example, a ...
TinyMotion: camera phone based interaction methods
CHI EA '06: CHI '06 Extended Abstracts on Human Factors in Computing Systems

This paper presents TinyMotion, a pure software approach that detects the movements of cell phones in real time by analyzing image sequences captured by the built-in camera. Typical movements that TinyMotion detects include - horizontal and vertical ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CCS '11: Proceedings of the 18th ACM conference on Computer and communications security

October 2011

742 pages

ISBN:9781450309486

DOI:10.1145/2046707

General Chair:
Yan Chen
Northwestern University, USA
,
Program Chairs:
George Danezis
Microsoft Research Cambridge, UK
,
Vitaly Shmatikov
University of Texas at Austin, USA

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSAC: ACM Special Interest Group on Security, Audit, and Control

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CCS'11

Sponsor:

SIGSAC

CCS'11: the ACM Conference on Computer and Communications Security

October 17 - 21, 2011

Illinois, Chicago, USA

Acceptance Rates

CCS '11 Paper Acceptance Rate 60 of 429 submissions, 14%;

Overall Acceptance Rate 1,261 of 6,999 submissions, 18%

Upcoming Conference

CCS '25

Sponsor:
sigsac

ACM SIGSAC Conference on Computer and Communications Security

October 13 - 17, 2025

Taipei , Taiwan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

69
Total Citations
View Citations
612
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)3

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Nassi BIluz ECohen OVayner ONassi DZadov BElovici Y(2024)Video-Based Cryptanalysis: Extracting Cryptographic Keys from Video Footage of a Device’s Power LED Captured by Standard Video Cameras2024 IEEE Symposium on Security and Privacy (SP)10.1109/SP54263.2024.00163(2422-2440)Online publication date: 19-May-2024
https://doi.org/10.1109/SP54263.2024.00163
Qiao HWang KHuang TXu XLiu SChen J(2024)A TEMPEST Attack Implementation based on Hidden Markov model in Smart GridJournal of Physics: Conference Series10.1088/1742-6596/2774/1/0120092774:1(012009)Online publication date: 1-Jul-2024
https://doi.org/10.1088/1742-6596/2774/1/012009
Farzand HAbraham MBrewster SKhamis MMarky K(2024)A Systematic Deconstruction of Human-Centric Privacy & Security Threats on Mobile PhonesInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2361519(1-24)Online publication date: 12-Jun-2024
https://doi.org/10.1080/10447318.2024.2361519
Lim JFrahm JMonrose FJoshi AFernandez MVerma R(2022)Leveraging Disentangled Representations to Improve Vision-Based Keystroke Inference Attacks Under Low Data ConstraintsProceedings of the Twelfth ACM Conference on Data and Application Security and Privacy10.1145/3508398.3511498(242-251)Online publication date: 14-Apr-2022
https://dl.acm.org/doi/10.1145/3508398.3511498
Cazorla JDe Fuentes JGonzález-Manzano L(2022)Eye-based keystroke prediction for natural texts – a feasibility analysis2022 IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom)10.1109/TrustCom56396.2022.00059(375-382)Online publication date: Dec-2022
https://doi.org/10.1109/TrustCom56396.2022.00059
Wei NSani A(2022)SchrodinText: Strong Protection of Sensitive Textual Content of Mobile ApplicationsIEEE Transactions on Mobile Computing10.1109/TMC.2020.302511921:4(1402-1419)Online publication date: 1-Apr-2022
https://doi.org/10.1109/TMC.2020.3025119
Sabra MMaiti AJadliwala M(2022)Background Buster: Peeking through Virtual Backgrounds in Online Video Calls2022 52nd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)10.1109/DSN53405.2022.00058(522-533)Online publication date: Jun-2022
https://doi.org/10.1109/DSN53405.2022.00058
Diamantaris MMoustakas SSun LIoannidis SPolakis JKim YKim JVigna GShi E(2021)This Sneaky Piggy Went to the Android Ad Market: Misusing Mobile Sensors for Stealthy Data ExfiltrationProceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security10.1145/3460120.3485366(1065-1081)Online publication date: 12-Nov-2021
https://dl.acm.org/doi/10.1145/3460120.3485366
Nassi BPirutin YGalor TElovici YZadov BKim YKim JVigna GShi E(2021)Glowworm Attack: Optical TEMPEST Sound Recovery via a Device's Power Indicator LEDProceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security10.1145/3460120.3484775(1900-1914)Online publication date: 12-Nov-2021
https://dl.acm.org/doi/10.1145/3460120.3484775
Jin WMurali SZhu HLi MKim YKim JVigna GShi E(2021)Periscope: A Keystroke Inference Attack Using Human Coupled Electromagnetic EmanationsProceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security10.1145/3460120.3484549(700-714)Online publication date: 12-Nov-2021
https://dl.acm.org/doi/10.1145/3460120.3484549
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents