research-article

PupilScreen: Using Smartphones to Assess Traumatic Brain Injury

Authors:

Alex Mariakakis,

Vardhman Mehta,

Megan A. Banks,

Shwetak N. PatelAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 1, Issue 3

Article No.: 81, Pages 1 - 27

https://doi.org/10.1145/3131896

Published: 11 September 2017 Publication History

Abstract

Before a person suffering from a traumatic brain injury reaches a medical facility, measuring their pupillary light reflex (PLR) is one of the few quantitative measures a clinician can use to predict their outcome. We propose PupilScreen, a smartphone app and accompanying 3D-printed box that combines the repeatability, accuracy, and precision of a clinical device with the ubiquity and convenience of the penlight test that clinicians regularly use in emergency situations. The PupilScreen app stimulates the patient's eyes using the smartphone's flash and records the response using the camera. The PupilScreen box, akin to a head-mounted virtual reality display, controls the eyes' exposure to light. The recorded video is processed using convolutional neural networks that track the pupil diameter over time, allowing for the derivation of clinically relevant measures. We tested two different network architectures and found that a fully convolutional neural network was able to track pupil diameter with a median error of 0.30 mm. We also conducted a pilot clinical evaluation with six patients who had suffered a TBI and found that clinicians were almost perfect when separating unhealthy pupillary light reflexes from healthy ones using PupilScreen alone.

Supplementary Material

mariakakis (mariakakis.zip)

Supplemental movie, appendix, image and software files for, PupilScreen: Using Smartphones to Assess Traumatic Brain Injury

Download
66.60 KB

References

[1]

Alireza Abdolvahabi, Brandon W. Taylor, Rebecca L. Holden, Elizabeth V Shaw, Alex Kentsis, Carlos Rodriguez-Galindo, Shizuo Mukai, and Bryan F. Shaw. Colorimetric and longitudinal analysis of leukocoria in recreational photographs of children with retinoblastoma. PloS one, 8(10):e76677, oct 2013.

[2]

Andrew Bastawrous, Hillary K. Rono, Iain A. T. Livingstone, Helen A. Weiss, Stewart Jordan, Hannah Kuper, and Matthew J. Burton. Development and Validation of a Smartphone-Based Visual Acuity Test (Peek Acuity) for Clinical Practice and Community-Based Fieldwork. JAMA Ophthalmology, 133(8):930, aug 2015.

[3]

Matthias Behrends, Claus U Niemann, and Merlin D Larson. Infrared pupillometry to detect the light reflex during cardiopulmonary resuscitation: A case series. Resuscitation, 83(10):1223--1228, 2012.

[4]

Barry P Boden, Robin L Tacchetti, Robert C Cantu, Sara B Knowles, and Frederick O Mueller. Catastrophic Head Injuries in High School and College Football Players. The American Journal of Sports Medicine, 35(7):1075--1081, mar 2007.

[5]

Steven P Broglio, Michael S Ferrara, Stephen N Macciocchi, Ted A Baumgartner, and Ronald Elliott. Test-retest reliability of computerized concussion assessment programs. Journal of Athletic Training, 42(4):509--514, 2007.

[6]

Jose E Capó-Aponté, Thomas G Urosevich, David V Walsh, Leonard A Temme, and Aaron K Tarbett. Pupillary Light Reflex as an Objective Biomarker for Early Identification of Blast-Induced mTBI. Journal of Spine, 2013.

[7]

Centers for Disease Control. TBI: Get the Facts, 2016.

[8]

Rama Chellappa. The changing fortunes of pattern recognition and computer vision. Image and Vision Computing, 55(1):3--5, 2016.

[9]

David Couret, Delphine Boumaza, Coline Grisotto, Thibaut Triglia, Lionel Pellegrini, Philippe Ocquidant, Nicolas J Bruder, and Lionel J Velly. Reliability of standard pupillometry practice in neurocritical care: an observational, double-blinded study. Critical Care, 20(1):99, dec 2016.

[10]

Mark Faul, Likang Xu, MM Wald, and VG Coronado. Traumatic Brain Injury in the United States. Technical report, 2010.

[11]

JD Fischer and DJ van den Heever. Portable video-oculography device for implementation in side-line concussion assessments: a prototype. In Proc. EMBC ‘16, 2016.

[12]

Wolfgang Fuhl, Thomas Kübler, Katrin Sippel, Wolfgang Rosenstiel, and Enkelejda Kasneci. ExCuSe: Robust Pupil Detection in Real-World Scenarios. In International Conference on Computer Analysis of Images and Patterns, pages 39--51. Springer International Publishing, 2015.

[13]

Wolfgang Fuhl, Thiago Santini, Gjergji Kasneci, and Enkelejda Kasneci. PupilNet: Convolutional Neural Networks for Robust Pupil Detection. (1-10), 2016.

[14]

Wolfgang Fuhl, Thiago C Santini, Thomas Kübler, and Enkelejda Kasneci. ElSe : Ellipse Selection for Robust Pupil Detection in Real-World Environments. In Eye Tracking Research 8 Applications, pages 123--130, 2016.

Digital Library

[15]

Kristin M Galetta, Lauren E Brandes, Karl Maki, Mark S Dziemianowicz, Eric Laudano, Megan Allen, Kathy Lawler, Brian Sennett, Douglas Wiebe, Steve Devick, Leonard V Messner, Steven L Galetta, and Laura J Balcer. Journal of the Neurological Sciences The King-Devick test and sports-related concussion : Study of a rapid visual screening tool in a collegiate cohort. Journal of the neurological sciences, 309(1-2):34--39, 2011.

[16]

Kristin M Galetta, Mengling Liu, Danielle F Leong, Rachel E Ventura, Steven L Galetta, and Laura J Balcer. The King-Devick test of rapid number naming for concussion detection: meta-analysis and systematic review of the literature. Concussion, 1(2):cnc.15.8, 2015.

[17]

Matthew S. Galetta, Kristin M. Galetta, Jim McCrossin, James A Wilson, Stephen Moster, Steven L Galetta, Laura J Balcer, Gary W Dorshimer, and Christina L Master. Saccades and memory: Baseline associations of the King-Devick and SCAT2 SAC tests in professional ice hockey players. Journal of the Neurological Sciences, 328(1-2):28--31, 2013.

[18]

Mario E Giardini, Iain A T Livingstone, Stewart Jordan, Nigel M Bolster, Tunde Peto, Matthew Burton, and Andrew Bastawrous. A smartphone based ophthalmoscope. Proc. EMBC '14, 2014:2177--2180, 2014.

[19]

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.

Digital Library

[20]

Kimberly G Harmon, Jonathan A Drezner, Matthew Gammons, Kevin M Guskiewicz, Mark Halstead, Stanley A Herring, Jeffrey S Kutcher, Andrea Pana, Margot Putukian, and William O Roberts. American Medical Society for Sports Medicine position statement: concussion in sport. British Journal of Sports Medicine, 47(1):15--26, jan 2013.

[21]

Moritz Kassner, William Patera, and Andreas Bulling. Pupil: an open source platform for pervasive eye tracking and mobile gaze-based interaction. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing Adjunct Publication - UbiComp '14 Adjunct, pages 1151--1160, New York, New York, USA, 2014. ACM Press.

Digital Library

[22]

Jeff Klingner, Rakshit Kumar, and Pat Hanrahan. Measuring the task-evoked pupillary response with a remote eye tracker. Proceedings of the 2008 symposium on Eye tracking research 8 applications - ETRA '08, 1(212):69, 2008.

Digital Library

[23]

Merlin D Larson and Matthias Behrends. Portable Infrared Pupillometry. Anesthesia 8 Analgesia, 120(6):1242--1253, 2015.

[24]

Merlin D Larson and Isobel Muhiudeen. Pupillometric analysis of the 'absent light reflex'. Archives of neurology, 52(4):369--72, 1995.

[25]

Everett Lawson, Jason Boggess, Siddharth Khullar, Alex Olwal, Gordon Wetzstein, and Ramesh Raskar. Computational retinal imaging via binocular coupling and indirect illumination. In Proc. SIGGRAPH '12, page 51, 2012.

Digital Library

[26]

Hopin Lee, S John Sullivan, Anthony G Schneiders, Osman Hassan Ahmed, Arun Prasad Balasundaram, David Williams, Willem H Meeuwisse, and Paul McCrory. Smartphone and tablet apps for concussion road warriors (team clinicians): a systematic review for practical users. British journal of sports medicine, 49(8):1--2, apr 2014.

[27]

Dongheng Li, David Winfield, and Derrick J Parkhurst. Starburst: A hybrid algorithm for video-based eye tracking combining feature-based and model-based approaches. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops, volume 3, pages 79--79. IEEE, 2005.

Digital Library

[28]

Joiiiv R Lindsay. The significance of a positional nystagmus in otoneurological diagnosis. The Laryngoscope, 55(10):527--551, 1945.

[29]

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 07-12-June, pages 3431--3440, mar 2015.

[30]

Alex Mariakakis, Megan A. Banks, Lauren Phillipi, Lei Yu, James Taylor, and Shwetak N. Patel. Biliscreen: smartphone-based scleral jaundice monitoring for liver and pancreatic disorders. In Proceedings of the 2017 ACM Interactive, Mobile, Wearable, Ubiquitous Technologies. ACM, 2017.

Digital Library

[31]

F Martínez-Ricarte, A Castro, MA Poca, J Sahuquillo, L Expósito, M Arribas, and J Aparicio. Infrared pupillometry. Basic principles and their application in the non-invasive monitoring of neurocritical patients. Neurología (Barcelona, Spain), 28(1):41--51, 2013.

[32]

Jun Maruta, Stephanie W. Lee, Emily F. Jacobs, and Jamshid Ghajar. A unified science of concussion. Annals of the New York Academy of Sciences, 1208(1):58--66, oct 2010.

[33]

Jun Maruta, J L Tong, Stephanie W Lee, Zarah Iqbal, Alison Schonberger, and Jamshid Ghajar. EYE-TRAC: monitoring attention and utility for mTBI. Sensing Technologies for Global Health, Military Medicine, Disaster Response, and Environmental Monitoring Ii: And Biometric Technology for Human Identification Ix, 8371:11, 2012.

[34]

Alex Marvez. Players could try to beat concussion tests | FOX Sports, 2011.

[35]

Paul McCrory, Willem H Meeuwisse, Mark Aubry, Bob Cantu, Jiri Dvorak, Ruben Echemendia, Lars Engebretsen, Karen Johnson, Jeffery S Kutcher, Martin Raftery, Allen Sills, Brian W Benson, Gavin A Davis, Richard G Ellenbogen, Kevin Guskiewicz, Stanley A Herring, Grant L Iverson, Barry D Jordan, James Kissick, Michael McCrea, Andrew S McIntosh, David Maddocks, Michael Makdissi, Laura Purcell, Margot Putukain, Kathryn Schneider, Charles H Tator, and Michael Turner. Consensus Statement on Concussion in Sport: The 4th International Conference on Concussion in Sport Held in Zurick, November 2012. British Journal of Sports Medicine, 47(2):250--258, apr 2013.

[36]

Michele Meeker, Rose Du, Peter Bacchetti, Claudio M Privitera, Merlin D Larson, Martin C Holland, and Geoffrey Manley. Pupil examination: validity and clinical utility of an automated pupillometer. The Journal of neuroscience nursing : journal of the American Association of Neuroscience Nurses, 37(1):34--40, 2005.

[37]

Tim Morris, Paul Blenkhorn, and Farhan Zaidi. Blink detection for real-time eye tracking. Journal of Network and Computer Applications, 25(2):129--143, 2002.

Digital Library

[38]

Rosemarie Scolaro Moser, Grant L Iverson, Ruben J Echemendia, Mark R Lovell, Philip Schatz, Frank M Webbe, Ronald M Ruff, Jeffrey T Barth, Donna K. NAN Policy and Planning Committee, Shane S. Donna K. Broshek, Shane S. Bush, Sandra P. Koffler, Cecil R. Reynolds, Cheryl H. Silver, Sandra P. Koffler, Cecil R. Reynolds, and Cheryl H. Silver. Neuropsychological evaluation in the diagnosis and management of sports-related concussion. Archives of clinical neuropsychology : the official journal of the National Academy of Neuropsychologists, 22(8):909--16, nov 2007.

[39]

Dai-Wai M Olson, Sonja Stutzman, Ciji Saju, Margaret Wilson, Weidan Zhao, and Venkatesh Aiyagari. Interrater Reliability of Pupillary Assessments. Neurocritical Care, 24(2):251--257, apr 2016.

[40]

Vitor F Pamplona, Ankit Mohan, Manuel M Oliveira, and Ramesh Raskar. NETRA: interactive display for estimating refractive errors and focal range. ACM transactions on graphics (TOG), 29(4):77, 2010.

Digital Library

[41]

Vitor F Pamplona, Erick B Passos, Jan Zizka, Manuel M Oliveira, Everett Lawson, Esteban Clua, and Ramesh Raskar. Catra: cataract probe with a lightfield display and a snap-on eyepiece for mobile phones. In Proc. SIGGRAPH ‘11, pages 7--11, 2011.

[42]

Stephen M Pizer, E. Philip Amburn, John D Austin, Robert Cromartie, Ari Geselowitz, Trey Greer, Bart ter Haar Romeny, John B Zimmerman, and Karel Zuiderveld. Adaptive histogram equalization and its variations. Computer Vision, Graphics, and Image Processing, 39(3):355--368, 1987.

Digital Library

[43]

Sohail Rafiqi, Chatchai Wangwiwattana, Jasmine Kim, Ephrem Fernandez, Suku Nair, and Eric C. Larson. PupilWare: towards pervasive cognitive load measurement using commodity devices. In Proceedings of the 8th ACM International Conference on Pervasive Technologies Related to Assistive Environments - PETRA '15, pages 1--8, New York, New York, USA, 2015. ACM Press.

Digital Library

[44]

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.

[45]

Lech Swirski, Andreas Bulling, and Neil Dodgson. Robust real-time pupil tracking in highly off-axis images. Etra, pages 1--4, 2012.

Digital Library

[46]

William R Taylor, Jeff W Chen, Hal Meltzer, Thomas A Gennarelli, Cynthia Kelbch, Sharen Knowlton, Jenny Richardson, Matthew J Lutch, Azadeh Farin, Kathryn N Hults, and Lawrence F Marshall. Quantitative pupillometry, a new technology: normative data and preliminary observations in patients with acute head injury. Technical note. Journal of neurosurgery, 98(1):205--213, 2003.

[47]

Preethi Thiagarajan and Kenneth J Ciuffreda. Pupillary responses to light in chronic non-blast-induced mTBI. Brain Injury, 29(12):1420--1425, oct 2015.

[48]

Fabian Timm and Erhardt Barth. Accurate Eye Centre Localisation by Means of Gradients. In VISAPP, pages 125--130, 2011.

[49]

James Q Truong and Kenneth J Ciuffreda. Comparison of pupillary dynamics to light in the mild traumatic brain injury (mTBI) and normal populations. Brain Injury, 30(11):1378--1389, sep 2016.

[50]

James Q Truong and Kenneth J Ciuffreda. Objective Pupillary Correlates of Photosensitivity in the Normal and Mild Traumatic Brain Injury Populations. Military Medicine, 181(10):1382--1390, oct 2016.

[51]

James Q Truong and Kenneth J Ciuffreda. Quantifying pupillary asymmetry through objective binocular pupillometry in the normal and mild traumatic brain injury (mTBI) populations. Brain Injury, 30(11):1372--1377, sep 2016.

[52]

James Q Truong, Nabin R Joshi, and Kenneth J Ciuffreda. Influence of refractive error on pupillary dynamics in the normal and mild traumatic brain injury (mTBI) populations. Journal of Optometry, mar 2017.

[53]

H Kenneth Walker, W Dallas Hall, and J Willis Hurst. Clinical Methods. Butterworths, 1990.

[54]

E Wood, T Baltruaitis, X Zhang, Y Sugano, P Robinson, and A Bulling. Rendering of Eyes for Eye-Shape Registration and Gaze Estimation, 2015.

[55]

Erroll Wood and Andreas Bulling. Eyetab: Model-based gaze estimation on unmodified tablet computers. Etra, pages 3--6, 2014.

Digital Library

[56]

Sahar F Zafar and Jose I Suarez. Automated pupillometer for monitoring the critically ill patient: A critical appraisal, 2014.

Cited By

Sousa AMarques-Neves CVieira P(2024)Development of a Smartphone-Based System for Intrinsically Photosensitive Retinal Ganglion Cells Targeted Chromatic PupillometryBioengineering10.3390/bioengineering1103026711:3(267)Online publication date: 9-Mar-2024
https://doi.org/10.3390/bioengineering11030267
Adhikary RSadeh MBatra NGoel M(2024)JoulesEyeProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314227:4(1-29)Online publication date: 12-Jan-2024
https://doi.org/10.1145/3631422
Islam RBae S(2024)PupilSense: Detection of Depressive Episodes through Pupillary Response in the Wild2024 International Conference on Activity and Behavior Computing (ABC)10.1109/ABC61795.2024.10652166(01-13)Online publication date: 29-May-2024
https://doi.org/10.1109/ABC61795.2024.10652166
Show More Cited By

Index Terms

PupilScreen: Using Smartphones to Assess Traumatic Brain Injury
1. Applied computing
  1. Life and medical sciences
    1. Consumer health
2. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile devices
      1. Smartphones

Recommendations

BiliScreen: Smartphone-Based Scleral Jaundice Monitoring for Liver and Pancreatic Disorders

Pancreatic cancer has one of the worst survival rates amongst all forms of cancer because its symptoms manifest later into the progression of the disease. One of those symptoms is jaundice, the yellow discoloration of the skin and sclera due to the ...
Beyond gaze: preliminary analysis of pupil dilation and blink rates in an fMRI study of program comprehension
EMIP '18: Proceedings of the Workshop on Eye Movements in Programming

Researchers have been employing psycho-physiological measures to better understand program comprehension, for example simultaneous fMRI and eye tracking to validate top-down comprehension models. In this paper, we argue that there is additional value in ...
Eye tracking as an MT evaluation technique

Eye tracking has been used successfully as a technique for measuring cognitive load in reading, psycholinguistics, writing, language acquisition etc. for some time now. Its application as a technique for measuring the reading ease of MT output has not ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 1, Issue 3

September 2017

2023 pages

EISSN:2474-9567

DOI:10.1145/3139486

Issue’s Table of Contents

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Received: 01 November 2017

Published: 11 September 2017

Accepted: 01 June 2017

Revised: 01 May 2017

Published in IMWUT Volume 1, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National Science Foundation Graduate Research Fellowship
ARCS Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

33
Total Citations
View Citations
688
Total Downloads

Downloads (Last 12 months)121
Downloads (Last 6 weeks)19

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sousa AMarques-Neves CVieira P(2024)Development of a Smartphone-Based System for Intrinsically Photosensitive Retinal Ganglion Cells Targeted Chromatic PupillometryBioengineering10.3390/bioengineering1103026711:3(267)Online publication date: 9-Mar-2024
https://doi.org/10.3390/bioengineering11030267
Adhikary RSadeh MBatra NGoel M(2024)JoulesEyeProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314227:4(1-29)Online publication date: 12-Jan-2024
https://doi.org/10.1145/3631422
Islam RBae S(2024)PupilSense: Detection of Depressive Episodes through Pupillary Response in the Wild2024 International Conference on Activity and Behavior Computing (ABC)10.1109/ABC61795.2024.10652166(01-13)Online publication date: 29-May-2024
https://doi.org/10.1109/ABC61795.2024.10652166
Maxin AKush SGulek BWinston GChae JShaibani RMcGrath LAbecassis ILevitt M(2024)Smartphone pupillometry for detection of cerebral vasospasm after aneurysmal subarachnoid hemorrhageJournal of Stroke and Cerebrovascular Diseases10.1016/j.jstrokecerebrovasdis.2024.10792233:10(107922)Online publication date: Oct-2024
https://doi.org/10.1016/j.jstrokecerebrovasdis.2024.107922
Maxin AGulek BLee CLim DMariakakis ALevitt MMcGrath L(2023)Validation of a Smartphone Pupillometry Application in Diagnosing Severe Traumatic Brain InjuryJournal of Neurotrauma10.1089/neu.2022.051640:19-20(2118-2125)Online publication date: 1-Oct-2023
https://doi.org/10.1089/neu.2022.0516
Barry CWang E(2023)Racially fair pupillometry measurements for RGB smartphone cameras using the far red spectrumScientific Reports10.1038/s41598-023-40796-013:1Online publication date: 24-Aug-2023
https://doi.org/10.1038/s41598-023-40796-0
Maxin AGulek BChae JWinston GWeisbeek PMcGrath LLevitt M(2023)A smartphone pupillometry tool for detection of acute large vessel occlusionJournal of Stroke and Cerebrovascular Diseases10.1016/j.jstrokecerebrovasdis.2023.10743032:12(107430)Online publication date: Dec-2023
https://doi.org/10.1016/j.jstrokecerebrovasdis.2023.107430
Saha PZulfiker SHashem TIslam K(2023)Dehydration Scan: An Artificial Intelligence Assisted Smartphone-Based System for Early Detection of DehydrationPervasive Computing Technologies for Healthcare10.1007/978-3-031-34586-9_20(289-312)Online publication date: 11-Jun-2023
https://doi.org/10.1007/978-3-031-34586-9_20
Solyman OAbushanab MCarey AHenderson A(2022)Pilot Study of Smartphone Infrared Pupillography and PupillometryClinical Ophthalmology10.2147/OPTH.S331989Volume 16(303-310)Online publication date: Feb-2022
https://doi.org/10.2147/OPTH.S331989
Barry Cde Souza JXuan YHolden JGranholm EWang E(2022)At-Home Pupillometry using Smartphone Facial Identification CamerasProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502493(1-12)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3502493
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents