Investigating Sensory Extensions as Input for Interactive Simulations
Authors: 
Chris Hill, Casey Lee Hunt, Sammie Crowder, Brett Fiedler, Emily B. Moore, Ann EisenbergAuthors Info & Claims
Chris Hill, Casey Lee Hunt, Sammie Crowder, Brett Fiedler, Emily B. Moore, Ann EisenbergAuthors Info & ClaimsArticle No.: 39, Pages 1 - 7
Abstract
Sensory extensions enhance our awareness by transforming variations in stimuli normally undetectable by human senses into perceivable outputs. Similarly, interactive simulations for learning promote an understanding of abstract phenomena. Combining sensory extension devices with interactive simulations gives users the novel opportunity to connect their sensory experiences in the physical world to computer-simulated concepts. We explore this opportunity by designing a suite of wearable sensory extension devices that interface with a uniquely inclusive PhET Simulation, Ratio and Proportion. In this simulation, two hands can be moved on-screen to various values, representing different mathematical ratios. Users explore changing hand heights to find and maintain ratios through visual and auditory feedback. Our sensory extension devices translate force, distance, sound frequency, and magnetic field strength to quantitative values in order to control individual hands in the computer simulation. This paper describes the design of the devices and our analysis of feedback from 23 high-school aged youth who used our designs to interact with the Ratio and Proportion simulation.
References
[1]
[n. d.]. AAH002-02E: 0.6 mt sat. Analog sensor, 2 Kohm, RoHS SOIC, 150c. https://www.nve.com/webstore/catalog/product_info.php?cPath=27_28&products_id=509
[2]
[n. d.]. Ear V1 3D model. https://free3d.com/3d-model/ear-v1–113169.html
[3]
[n. d.]. Ratio and proportion. https://phet.colorado.edu/en/simulations/ratio-and-proportion/
[4]
[n. d.]. Sparkfun analog MEMS microphone breakout - ICS-40180. https://www.sparkfun.com/products/18011
[5]
[n. d.]. SparkFun USB to serial breakout - FT232RL. https://www.sparkfun.com/products/12731
[6]
2011. GP. https://www.amazon.com/gp/product/B08BL4PBSZ/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1
[7]
Sriram Karthik Badam, Arjun Srinivasan, and Niklas Elmqvist. 2017. Affordances of Input Modalities for Visual Data Exploration in Immersive Environments.
[8]
Rafael Ballagas, Meredith Ringel, Maureen Stone, and Jan Borchers. 2003. IStuff: A Physical User Interface Toolkit for Ubiquitous Computing Environments. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’03). Association for Computing Machinery, New York, NY, USA, 537–544. https://doi.org/10.1145/642611.642705
[9]
Diana Carvalho, Maximino Bessa, Luís Magalhães, and Eurico Carrapatoso. 2016. Age Group Differences in Performance Using Diverse Input Modalities: Insertion Task Evaluation. In Proceedings of the XVII International Conference on Human Computer Interaction(Interacción ’16). Association for Computing Machinery, New York, NY, USA, Article 12, 8 pages. https://doi.org/10.1145/2998626.2998664
[10]
Alvaro Cassinelli, Carson Reynolds, and Masatoshi Ishikawa. 2006. Augmenting spatial awareness with Haptic Radar. In 2006 10th IEEE International Symposium on Wearable Computers. 61–64. https://doi.org/10.1109/ISWC.2006.286344
[11]
Julia Chatain, Marie Demangeat, Anke M. Brock, Didier Laval, and Martin Hachet. 2015. Exploring Input Modalities for Interacting with Augmented Paper Maps. In Proceedings of the 27th Conference on l’Interaction Homme-Machine(IHM ’15). Association for Computing Machinery, New York, NY, USA, Article 22, 6 pages. https://doi.org/10.1145/2820619.2825002
[12]
Alois Ferscha, Bernadette Emsenhuber, Andreas Riener, Clemens Holzmann, Manfred Hechinger, Dominik Hochreiter, Marquart Franz, Andreas Zeidler, and Marcos dos Santos Rocha. 2008. Vibro-Tactile Space-Awareness.
[13]
Brett L. Fiedler, Taliesin L. Smith, Jesse Greenberg, and Emily B. Moore. 2022. For One or for All? Survey of Educator Perceptions of Web Speech-Based Auditory Description in Science Interactives. In Proceedings of the 19th International Web for All Conference(W4A ’22). Association for Computing Machinery, New York, NY, USA, Article 5, 11 pages. https://doi.org/10.1145/3493612.3520456
[14]
Virginia Flood, Anna Shvarts, and Dor Abrahamson. 2020. Teaching with embodied learning technologies for mathematics: responsive teaching for embodied learning. ZDM 52 (07 2020). https://doi.org/10.1007/s11858-020-01165-7
[15]
Heng Gu, Kai Kunze, Masashi Takatani, and Kouta Minamizawa. 2015. Towards Performance Feedback through Tactile Displays to Improve Learning Archery. In Adjunct Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers(UbiComp/ISWC’15 Adjunct). Association for Computing Machinery, New York, NY, USA, 141–144. https://doi.org/10.1145/2800835.2800893
[16]
Yoni Halperin, Galit Buchs, Shachar Maidenbaum, Maya Amenou, and Amir Amedi. 2016. Social Sensing: A Wi-Fi Based Social Sense for Perceiving the Surrounding People. In Proceedings of the 7th Augmented Human International Conference 2016(AH ’16). Association for Computing Machinery, New York, NY, USA, Article 42, 2 pages. https://doi.org/10.1145/2875194.2875228
[17]
Abraham M. Hashemian, Matin Lotfaliei, Ashu Adhikari, Ernst Kruijff, and Bernhard E. Riecke. 2022. HeadJoystick: Improving Flying in VR Using a Novel Leaning-Based Interface. IEEE Transactions on Visualization and Computer Graphics 28, 4(2022), 1792–1809. https://doi.org/10.1109/TVCG.2020.3025084
[18]
Chris Hill, Michael Schneider, Mark Gross, Ann Eisenberg, Arielle Blum, and Mark Gross. 2020. A Wearable Meter That Actively Monitors the Continuity of E- Textile Circuits as They Are Sewn. https://doi.org/10.1145/3386201.3386217
[19]
Adafruit Industries. [n. d.]. Adafruit VL53L4CD time of flight distance sensor - 1 to 1300mm. https://www.adafruit.com/product/5396
[20]
Karim Jebari. 2015. Sensory Enhancement. (01 2015), 827–838. https://doi.org/10.1007/978-94-007-4707-4_106
[21]
Annie Kelly, Christine Chang, Christian Hill, Mary West, Mary Yoder, Joseph Polman, Shaun Kane, Michael Eisenberg, and R. Benjamin Shapiro. [n. d.]. “Our Dog Probably Thinks Christmas is Really Boring”: Re-mediating Science Education for Feminist-Inspired Inquiry. Proceedings of International Conference of the Learning Sciences (ICLS) 2020 ([n. d.]). https://doi.org/10.22318/icls2020.935
[22]
David Lakatos, Matthew Blackshaw, Alex Olwal, Zachary Barryte, Ken Perlin, and Hiroshi Ishii. 2014. T(Ether): Spatially-Aware Handhelds, Gestures and Proprioception for Multi-User 3D Modeling and Animation. In Proceedings of the 2nd ACM Symposium on Spatial User Interaction(SUI ’14). Association for Computing Machinery, New York, NY, USA, 90–93. https://doi.org/10.1145/2659766.2659785
[23]
Scott Lambert, Brett Fiedler, Chloe Hershenow, Dor Abrahamson, Jenna Gorlewicz, and Aaron Cobian. 2022. A Tangible Manipulative for Inclusive Quadrilateral Learning. (03 2022), 16.
[24]
Joseph J. LaViola, Daniel Acevedo Feliz, Daniel F. Keefe, and Robert C. Zeleznik. 2001. Hands-Free Multi-Scale Navigation in Virtual Environments. In Proceedings of the 2001 Symposium on Interactive 3D Graphics(I3D ’01). Association for Computing Machinery, New York, NY, USA, 9–15. https://doi.org/10.1145/364338.364339
[25]
Louis Longin and Ophelia Deroy. 2022. Augmenting perception: How artificial intelligence transforms sensory substitution. Consciousness and Cognition 99 (2022), 103280. https://doi.org/10.1016/j.concog.2022.103280
[26]
Andrés Lucero. 2015. Using Affinity Diagrams to Evaluate Interactive Prototypes, Vol. 9297. 231–248. https://doi.org/10.1007/978-3-319-22668-2_19
[27]
Leilah Lyons, Brian Slattery, Priscilla Jimenez, Brenda Lopez, and Tom Moher. 2012. Don’t Forget about the Sweat: Effortful Embodied Interaction in Support of Learning. In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction(TEI ’12). Association for Computing Machinery, New York, NY, USA, 77–84. https://doi.org/10.1145/2148131.2148149
[28]
Saskia K. Nagel, Christine Carl, Tobias Kringe, Robert Märtin, and Peter König. 2005. Beyond sensory substitution—learning the sixth sense. Journal of Neural Engineering 2 (2005), R13 – R26.
[29]
Leyla Norooz, Matthew Louis Mauriello, Anita Jorgensen, Brenna McNally, and Jon E. Froehlich. 2015. BodyVis: A New Approach to Body Learning Through Wearable Sensing and Visualization. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems(CHI ’15). Association for Computing Machinery, New York, NY, USA, 1025–1034. https://doi.org/10.1145/2702123.2702299
[30]
Noriyasu Obushi, Sohei Wakisaka, Shunichi Kasahara, Katie Seaborn, Atsushi Hiyama, and Masahiko Inami. 2019. MagniFinger: Fingertip Probe Microscope with Direct Micro Movements. In Proceedings of the 10th Augmented Human International Conference 2019(AH2019). Association for Computing Machinery, New York, NY, USA, Article 32, 7 pages. https://doi.org/10.1145/3311823.3311859
[31]
Katherine Perkins and Emily Moore. 2017. Increasing the accessibility of PhET Simulations for students with disabilities: Progress, challenges, and potential. In Physics Education Research Conference 2017(PER Conference). Cincinnati, OH, 296–299.
[32]
Rosalind W. Picard and Jennifer Healey. 1997. Affective wearables. Personal Technologies 1(1997), 231–240.
[33]
Yvonne Rogers, Sara Price, Geraldine Fitzpatrick, Rowanne Fleck, Eric Harris, Hilary Smith, Cliff Randell, Henk Muller, Claire O’Malley, Danäe Stanton, Mark Thompson, and Mark Weal. 2004. Ambient Wood: Designing New Forms of Digital Augmentation for Learning Outdoors. (01 2004). https://doi.org/10.1145/1017833.1017834
[34]
Ayelet Segal. 2011. Do Gestural Interfaces Promote Thinking? Embodied Interaction: Congruent Gestures and Direct Touch Promote Performance in Math. (01 2011).
[35]
Taliesin L. Smith and Emily B. Moore. 2020. Storytelling to Sensemaking: A Systematic Framework for Designing Auditory Description Display for Interactives. Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376460
[36]
Sofia Tancredi, Rotem Abdu, Dor Abrahamson, and Ramesh Balasubramaniam. 2021. Modeling nonlinear dynamics of fluency development in an embodied-design mathematics learning environment with Recurrence Quantification Analysis. International Journal of Child-Computer Interaction 29 (2021), 100297. https://doi.org/10.1016/j.ijcci.2021.100297
[37]
Sofia Tancredi, Julia Wang, Helen Tong Li, Carissa Jiayuan Yao, Genna Macfarlan, and Kimiko Ryokai. 2022. Balance Board Math: “Being the Graph” through the Sense of Balance for Embodied Self-Regulation and Learning. In Interaction Design and Children(IDC ’22). Association for Computing Machinery, New York, NY, USA, 137–149. https://doi.org/10.1145/3501712.3529743
[38]
J. L. Tennison, J. Greenberg, E. B. Moore, and J. L. & Gorlewicz. [n. d.]. Haptic paradigms for multimodal interactive simulations. Journal on technology and persons with disabilities 9 ([n. d.]). https://par.nsf.gov/biblio/10318226
[39]
Thingiverse.com. [n. d.]. 50kg loadcell bracket (with fusion 360 design) by IXP. https://www.thingiverse.com/thing:4856495
[40]
Naomi Thompson, Kylie Peppler, and Joshua Danish. 2017. Designing BioSim: Playfully Encouraging Systems Thinking in Young Children. 149–167. https://doi.org/10.4018/978-1-5225-1837-2.ch018
[41]
Brianna J. Tomlinson, Bruce N. Walker, and Emily B. Moore. 2020. Auditory Display in Interactive Science Simulations: Description and Sonification Support Interaction and Enhance Opportunities for Learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems(CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376886
[42]
WebBluetoothCG. 2022. Web-bluetooth/implementation-status.md at main · webbluetoothcg/web-bluetooth. https://github.com/WebBluetoothCG/web-bluetooth/blob/main/implementation-status.md#notes
[43]
ZackFreedman. [n. d.]. Zackfreedman/thunder-finger: Feel electric current... with your finger.https://github.com/ZackFreedman/THUNDER-FINGER
Index Terms
- Investigating Sensory Extensions as Input for Interactive Simulations
Recommendations
Interactive Processing of Sensory Input and Motor Output in the Human Hippocampus
Recent studies of visuomotor integration suggest that the motor system may be intimately involved in the detection of salient features of the sensory scene. The final stages of sensory processing occur in hippocampal structures. We measured human ...
Attention and Sensory Interactions within the Occipital Cortex in the Early Blind: An fMRI Study
Visual deprivation early in life results in occipital cortical responsiveness across a broad range of perceptual and cognitive tasks. In the reorganized occipital cortex of early blind (EB) individuals, the relative lack of specificity for particular ...
Tickling Expectations: Neural Processing in Anticipation of a Sensory Stimulus
Predictions of the near future can optimize the accuracy and speed of sensory processing as well as of behavioral responses. Previous experience and contextual cues are essential elements in the generation of a subjective prediction. Using a blocked ...
Comments
Information & Contributors
Information
Published In
February 2023
709 pages
ISBN:9781450399777
DOI:10.1145/3569009
Copyright © 2023 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 26 February 2023
Check for updates
Author Tags
Qualifiers
- Work in progress
- Research
- Refereed limited
Funding Sources
Conference
TEI '23
Sponsor:
TEI '23: Seventeenth International Conference on Tangible, Embedded, and Embodied Interaction
February 26 - March 1, 2023
Warsaw, Poland
Acceptance Rates
Overall Acceptance Rate 393 of 1,367 submissions, 29%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 167Total Downloads
- Downloads (Last 12 months)99
- Downloads (Last 6 weeks)26
Reflects downloads up to 06 Oct 2024
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatGet Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in