Work in Progress

Project Mihr: Enabling Gestural Interactions on a Keyboard using a Graphene-based Fabric

Authors:

Evelina Barhudarian,

Teddy SeyedAuthors Info & Claims

CHI EA '23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems

Article No.: 236, Pages 1 - 7

https://doi.org/10.1145/3544549.3585873

Published: 19 April 2023 Publication History

Abstract

The physical keyboard has long been the de facto input mechanism for computers, despite advances in other input forms and modalities. Even with these advances, the augmentations of physical keyboards have still been limited to keystroke-based entry. We leverage the inevitable physical touch and motions over a keyboard during keystroke-based input to present Project Mihr, a physical keyboard that has been augmented to be touch-sensitive. We leveraged passive capacitive sensing with a graphene-based textile to detect and classify rich and expressive gestural interactions over a keyboard. We conclude this work by describing our characterization on the sensing performance and presenting several different application scenarios that are enabled by our technique over keyboards.

Supplementary Material

MP4 File (3544549.3585873-video-figure.mp4)

Video Figure

Download
177.57 MB

MP4 File (3544549.3585873-video-preview.mp4)

Video Preview

Download
28.00 MB

MP4 File (3544549.3585873-talk-video.mp4)

Pre-recorded Video Presentation

Download
109.66 MB

References

[1]

2013. Logitech FabricSkin Keyboard Folio for iPad (review) | ZDNET. [Online]. Available from: https://www.zdnet.com/article/logitech-fabricskin-keyboard-folio-for-ipad-review/.

[2]

2019. Alcantara fabric on Microsoft Surface PCs: Everything you need to know | Windows Central. [Online]. Available from: https://www.windowscentral.com/what-is-alcantara-microsoft-surface.

[3]

2019. Alcantara fabric: What is it and is it worth it? | carwow. [Online]. Available from: https://www.carwow.co.uk/guides/glossary/what-is-alcantara-0518.

[4]

2022. Amazon.com: Echo (2nd Generation) International Version – Charcoal Fabric : Amazon Devices & Accessories. [Online]. Available from: https://www.amazon.com/Echo-2nd-Generation-International-Version/dp/B075RSCZHD.

[5]

Roland Aigner, Andreas Pointner, Thomas Preindl, Rainer Danner, and Michael Haller. 2021. TexYZ: Embroidering Enameled Wires for Three Degree-of-Freedom Mutual Capacitive Sensing. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 499, 12 pages. https://doi.org/10.1145/3411764.3445479

Digital Library

[6]

Roland Aigner, Andreas Pointner, Thomas Preindl, Patrick Parzer, and Michael Haller. 2020. Embroidered resistive pressure sensors: A novel approach for textile interfaces. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.

Digital Library

[7]

Daniel Buschek, Bianka Roppelt, and Florian Alt. 2018. Extending keyboard shortcuts with arm and wrist rotation gestures. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–12.

Digital Library

[8]

Nalin Chhibber, Hemant Bhaskar Surale, Fabrice Matulic, and Daniel Vogel. 2021. Typealike: Near-Keyboard Hand Postures for Expanded Laptop Interaction. Proceedings of the ACM on Human-Computer Interaction 5, ISS (2021), 1–20.

Digital Library

[9]

Tobias Grosse-Puppendahl, Christian Holz, Gabe Cohn, Raphael Wimmer, Oskar Bechtold, Steve Hodges, Matthew S Reynolds, and Joshua R Smith. 2017. Finding common ground: A survey of capacitive sensing in human-computer interaction. In Proceedings of the 2017 CHI conference on human factors in computing systems. 3293–3315.

Digital Library

[10]

Nur Al-huda Hamdan, Simon Voelker, and Jan Borchers. 2018. Sketch&Stitch: Interactive Embroidery for E-Textiles. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3173574.3173656

Digital Library

[11]

Cedric Honnet, Hannah Perner-Wilson, Marc Teyssier, Bruno Fruchard, Jürgen Steimle, Ana C. Baptista, and Paul Strohmeier. 2020. PolySense: Augmenting Textiles with Electrical Functionality Using In-Situ Polymerization. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13.

Digital Library

[12]

Seung-Bae Jeon, Sang-Jae Park, Weon-Guk Kim, Il-Woong Tcho, Ik-Kyeong Jin, Joon-Kyu Han, Daewon Kim, and Yang-Kyu Choi. 2018. Self-powered wearable keyboard with fabric based triboelectric nanogenerator. Nano energy 53 (2018), 596–603.

[13]

Hye Ran Koo, Young-Jae Lee, Sunok Gi, Seonah Khang, Joo Hyeon Lee, Jae-Ho Lee, Min-Gyu Lim, Hee-Jung Park, and Jeong-Whan Lee. 2014. The Effect of Textile-Based Inductive Coil Sensor Positions for Heart Rate Monitoring. J. Med. Syst. 38, 2 (feb 2014), 1–12. https://doi.org/10.1007/s10916-013-0002-0

Digital Library

[14]

Joanne Leong, Patrick Parzer, Florian Perteneder, Teo Babic, Christian Rendl, Anita Vogl, Hubert Egger, Alex Olwal, and Michael Haller. 2016. proCover: sensory augmentation of prosthetic limbs using smart textile covers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 335–346.

Digital Library

[15]

Yiyue Luo, Yunzhu Li, Pratyusha Sharma, Wan Shou, Kui Wu, Michael Foshey, Beichen Li, Tomás Palacios, Antonio Torralba, and Wojciech Matusik. 2021. Learning human–environment interactions using conformal tactile textiles. Nature Electronics 4, 3 (2021), 193–201.

[16]

Yiyue Luo, Kui Wu, Tomás Palacios, and Wojciech Matusik. 2021. KnitUI: Fabricating interactive and sensing textiles with machine knitting. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–12.

Digital Library

[17]

Yiyue Luo, Kui Wu, Andrew Spielberg, Michael Foshey, Daniela Rus, Tomás Palacios, and Wojciech Matusik. 2022. Digital Fabrication of Pneumatic Actuators with Integrated Sensing by Machine Knitting. In CHI Conference on Human Factors in Computing Systems. 1–13.

Digital Library

[18]

Charles McLellan. 2012. Eleksen Wireless Fabric Keyboard: a first look. Retrieved from: on May 7 (2012).

[19]

Maggie Orth, Joshua R Smith, E Rehmi Post, JA Strickon, and Emily B Cooper. 1998. Musical jacket. In ACM SIGGRAPH 98 Electronic art and animation catalog. 38.

[20]

Alexander Otte, Tim Menzner, Travis Gesslein, Philipp Gagel, Daniel Schneider, and Jens Grubert. 2019. Towards utilizing touch-sensitive physical keyboards for text entry in virtual reality. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 1729–1732.

[21]

Jifei Ou, Daniel Oran, Don Derek Haddad, Joseph Paradiso, and Hiroshi Ishii. 2019. SensorKnit: Architecting textile sensors with machine knitting. 3D Printing and Additive Manufacturing 6, 1 (2019), 1–11.

[22]

Patrick Parzer, Adwait Sharma, Anita Vogl, Jürgen Steimle, Alex Olwal, and Michael Haller. 2017. SmartSleeve: real-time sensing of surface and deformation gestures on flexible, interactive textiles, using a hybrid gesture detection pipeline. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. 565–577.

Digital Library

[23]

Andreas Pointner, Thomas Preindl, Sara Mlakar, Roland Aigner, and Michael Haller. 2020. Knitted RESi: A Highly Flexible, Force-Sensitive Knitted Textile Based on Resistive Yarns. In ACM SIGGRAPH 2020 Emerging Technologies (Virtual Event, USA) (SIGGRAPH ’20). Association for Computing Machinery, New York, NY, USA, Article 21, 2 pages.

Digital Library

[24]

Ivan Poupyrev, Nan-Wei Gong, Shiho Fukuhara, Mustafa Emre Karagozler, Carsten Schwesig, and Karen E. Robinson. 2016. Project Jacquard: Interactive Digital Textiles at Scale. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 4216–4227. https://doi.org/10.1145/2858036.2858176

Digital Library

[25]

Ivan Poupyrev, Nan-Wei Gong, Shiho Fukuhara, Mustafa Emre Karagozler, Carsten Schwesig, and Karen E. Robinson. 2016. Project Jacquard: Interactive Digital Textiles at Scale. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 4216–4227.

Digital Library

[26]

Julian Ramos, Zhen Li, Johana Rosas, Nikola Banovic, Jennifer Mankoff, and Anind Dey. 2016. Keyboard surface interaction: Making the keyboard into a pointing device. arXiv preprint arXiv:1601.04029 (2016).

[27]

Stefan Schneegass and Alexandra Voit. 2016. GestureSleeve: Using Touch Sensitive Fabrics for Gestural Input on the Forearm for Controlling Smartwatches. In Proceedings of the 2016 ACM International Symposium on Wearable Computers (Heidelberg, Germany) (ISWC ’16). Association for Computing Machinery, New York, NY, USA, 108–115.

Digital Library

[28]

Yilei Shi, Haimo Zhang, Hasitha Rajapakse, Nuwan Tharaka Perera, Tomás Vega Gálvez, and Suranga Nanayakkara. 2018. Gestakey: Touch interaction on individual keycaps. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–12.

Digital Library

[29]

Stuart Taylor, Cem Keskin, Otmar Hilliges, Shahram Izadi, and John Helmes. 2014. Type-hover-swipe in 96 bytes: A motion sensing mechanical keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1695–1704.

Digital Library

[30]

Ying-Chao Tung, Ta Yang Cheng, Neng-Hao Yu, Chiuan Wang, and Mike Y Chen. 2015. FlickBoard: enabling trackpad interaction with automatic mode switching on a capacitive-sensing keyboard. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 1847–1850.

Digital Library

[31]

Nicolas Villar, Daniel Cletheroe, Greg Saul, Christian Holz, Tim Regan, Oscar Salandin, Misha Sra, Hui-Shyong Yeo, William Field, and Haiyan Zhang. 2018. Project Zanzibar: A Portable and Flexible Tangible Interaction Platform. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, Article 515, 13 pages.

Digital Library

[32]

Paul X Wang, Chia Chi Wu, Qiliang Xu, Zheng Gao, Daniel D Sunshine, and Aidan N Zimmerman. 2019. Keyboard with touch sensor illumination. US Patent 10,514,772.

[33]

Irmandy Wicaksono and Joseph A Paradiso. 2017. Fabrickeyboard: multimodal textile sensate media as an expressive and deformable musical interface. In NIME, Vol. 17. 348–353.

[34]

Andrew D Wilson. 2006. Robust computer vision-based detection of pinching for one and two-handed gesture input. In Proceedings of the 19th annual ACM symposium on User interface software and technology. 255–258.

Digital Library

[35]

Jacob Wobbrock, Brad Myers, and Brandon Rothrock. 2006. Few-key text entry revisited: mnemonic gestures on four keys. In Proceedings of the SIGCHI conference on Human Factors in computing systems. 489–492.

Digital Library

[36]

Te-Yen Wu, Lu Tan, Yuji Zhang, Teddy Seyed, and Xing-Dong Yang. 2020. Capacitivo: Contact-based object recognition on interactive fabrics using capacitive sensing. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 649–661.

Digital Library

[37]

Te-Yen Wu, Zheer Xu, Xing-Dong Yang, Steve Hodges, and Teddy Seyed. 2021. Project Tasca: Enabling Touch and Contextual Interactions with a Pocket-Based Textile Sensor. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 4, 13 pages. https://doi.org/10.1145/3411764.3445712

Digital Library

[38]

Haimo Zhang and Yang Li. 2014. Gestkeyboard: enabling gesture-based interaction on ordinary physical keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1675–1684.

Digital Library

[39]

Yang Zhang, Robert Xiao, and Chris Harrison. 2016. Advancing hand gesture recognition with high resolution electrical impedance tomography. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 843–850.

Digital Library

[40]

Jingjie Zheng and Daniel Vogel. 2016. Finger-aware shortcuts. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 4274–4285.

Digital Library

[41]

Lara Zlokapa, Yiyue Luo, Jie Xu, Michael Foshey, Kui Wu, Pulkit Agrawal, and Wojciech Matusik. 2022. An Integrated Design Pipeline for Tactile Sensing Robotic Manipulators. In 2022 International Conference on Robotics and Automation (ICRA). IEEE, 3136–3142.

Cited By

Luo YZhu JWu KHonnet CMueller SMatusik W(2023)MagKnitic: Machine-knitted Passive and Interactive Haptic Textiles with Integrated Binary SensingProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606765(1-13)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606765
Luo Y(2023)Intelligent Textiles for Physical Human-Environment InteractionsAdjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586182.3616704(1-5)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586182.3616704

Index Terms

Project Mihr: Enabling Gestural Interactions on a Keyboard using a Graphene-based Fabric
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Keyboards

Recommendations

Touchless gestural interaction with small displays: a case study
CHItaly '13: Proceedings of the Biannual Conference of the Italian Chapter of SIGCHI

Touchless gestural interaction enables users to interact with digital devices using body movements and gestures, and without the burden of a physical contact with technology (e.g., data gloves, body markers, or remote controllers). Most gesture-based ...
Tangible Meets Gestural: Comparing and Blending Post-WIMP Interaction Paradigms
TEI '15: Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction

More and more objects of our everyday environment are becoming smart and connected, offering us new interaction possibilities. Tangible interaction and gestural interaction are promising communication means with these objects in this post-WIMP ...
Novel Interactions on the Keyboard

A new type of device retains the physicality of the conventional keyboard but includes extensions with touch sensing and dynamic graphics output that enable novel interactions with computers. The featured Web extra is a video of authors Hans Gellersen ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI EA '23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems

April 2023

3914 pages

ISBN:9781450394222

DOI:10.1145/3544549

Editors:
Albrecht Schmidt
LMU Munich, Germany
,
Kaisa Väänänen
Tampere University, Finland
,
Tesh Goyal
Google Research, USA
,
Per Ola Kristensson
University of Cambridge, UK
,
Anicia Peters
University of Namibia, Namibia

Copyright © 2023 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 April 2023

Check for updates

Author Tags

Qualifiers

Work in progress
Research
Refereed limited

Conference

CHI '23

Sponsor:

SIGCHI

CHI '23: CHI Conference on Human Factors in Computing Systems

April 23 - 28, 2023

Hamburg, Germany

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
278
Total Downloads

Downloads (Last 12 months)137
Downloads (Last 6 weeks)21

Reflects downloads up to 11 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Luo YZhu JWu KHonnet CMueller SMatusik W(2023)MagKnitic: Machine-knitted Passive and Interactive Haptic Textiles with Integrated Binary SensingProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606765(1-13)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606765
Luo Y(2023)Intelligent Textiles for Physical Human-Environment InteractionsAdjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586182.3616704(1-5)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586182.3616704

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Table of Contents