Inline Visualization and Manipulation of Real-Time Hardware Log for Supporting Debugging of Embedded Programs
Authors: 
Andrea Bianchi, 
Zhi Lin Yap, Punn Lertjaturaphat, Austin Z. Henley, 
Kongpyung Justin Moon, Yoonji KimAuthors Info & Claims
Andrea Bianchi, Zhi Lin Yap, Punn Lertjaturaphat, Austin Z. Henley, Kongpyung Justin Moon, Yoonji KimAuthors Info & ClaimsArticle No.: 248, Pages 1 - 26
Abstract
The advent of user-friendly embedded prototyping systems, exemplified by platforms like Arduino, has significantly democratized the creation of interactive devices that combine software programs with electronic hardware. This interconnection between hardware and software, however, makes the identification of bugs very difficult, as problems could be rooted in the program, in the circuit, or at their intersection. While there are tools to assist in identifying and resolving bugs, they typically require hardware instrumentation or visualizing logs in serial monitors. Based on the findings of a formative study, we designed Inline a programming tool that simplifies debugging of embedded systems by making explicit the internal state of the hardware and the program's execution flow using visualizations of the hardware logs directly within the user's code. The system's key characteristics are 1) an inline presentation of logs within the code, 2) real-time tracking of the execution flow, and 3) an expression language to manipulate and filter the logs. The paper presents the detailed implementation of the system and a study with twelve users, which demonstrates what features were adopted and how they were leveraged to complete debugging tasks.
Supplemental Material
MP4 File
Demonstration video
- Download
- 15.94 MB
References
[1]
Andrea Bianchi, Kongpyung Justin Moon, Artem Dementyev, and Seungwoo Je. 2024. BlinkBoard: Guiding and monitoring circuit assembly for synchronous and remote physical computing education. HardwareX (2024), e00511. https://doi.org/10.1016/j.ohx.2024.e00511
[2]
Tracey Booth, Simone Stumpf, Jon Bird, and Sara Jones. 2016. Crossed Wires: Investigating the Problems of End-User Developers in a Physical Computing Task. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI '16). Association for Computing Machinery, New York, NY, USA, 3485--3497. https://doi.org/10.1145/2858036.2858533
[3]
Anke Brocker, Simon Voelker, Tony Zelun Zhang, Mathis Müller, and Jan Borchers. 2019. Flowboard: A Visual Flow-Based Programming Environment for Embedded Coding. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA '19). Association for Computing Machinery, New York, NY, USA, 1--4. https://doi.org/10.1145/3290607.3313247
[4]
Brian Burg, Richard Bailey, Amy J. Ko, and Michael D. Ernst. 2013. Interactive Record/Replay for Web Application Debugging. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (St. Andrews, Scotland, United Kingdom) (UIST '13). Association for Computing Machinery, New York, NY, USA, 473--484. https://doi.org/10.1145/2501988.2502050
[5]
Alvaro Cassinelli and Daniel Saakes. 2017. Data Flow, Spatial Physical Computing. In Proceedings of the Eleventh International Conference on Tangible, Embedded, and Embodied Interaction (Yokohama, Japan) (TEI '17). Association for Computing Machinery, New York, NY, USA, 253--259. https://doi.org/10.1145/3024969.3024978
[6]
Ishan Chatterjee, Tadeusz Pforte, Aspen Tng, Farshid Salemi Parizi, Chaoran Chen, and Shwetak Patel. 2022. ARDW: An Augmented Reality Workbench for Printed Circuit Board Debugging. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (Bend, OR, USA) (UIST '22). Association for Computing Machinery, New York, NY, USA, Article 37, 16 pages. https://doi.org/10.1145/3526113.3545684
[7]
Josh Urban Davis, Jun Gong, Yunxin Sun, Parmit Chilana, and Xing-Dong Yang. 2019. CircuitStyle: A System for Peripherally Reinforcing Best Practices in Hardware Computing. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST '19). Association for Computing Machinery, New York, NY, USA, 109--120. https://doi.org/10.1145/3332165.3347920
[8]
James Devine, Michal Moskal, Peli de Halleux, Thomas Ball, Steve Hodges, Gabriele D'Amone, David Gakure, Joe Finney, Lorraine Underwood, Kobi Hartley, Paul Kos, and Matt Oppenheim. 2022. Plug-and-Play Physical Computing with Jacdac. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 6, 3, Article 110 (sep 2022), 30 pages. https://doi.org/10.1145/3550317
[9]
Daniel Drew, Julie L. Newcomb, William McGrath, Filip Maksimovic, David Mellis, and Björn Hartmann. 2016. The Toastboard: Ubiquitous Instrumentation and Automated Checking of Breadboarded Circuits. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). Association for Computing Machinery, New York, NY, USA, 677--686. https://doi.org/10.1145/2984511.2984566
[10]
Camille Gobert and Michel Beaudouin-Lafon. 2023. Lorgnette: Creating Malleable Code Projections. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST '23). Association for Computing Machinery, New York, NY, USA, Article 71, 16 pages. https://doi.org/10.1145/3586183.3606817
[11]
Cristina Guerrero-Romero, Simon Lucas, and Diego Perez-Liebana. 2023. Beyond Playing to Win: Creating a Team of Agents With Distinct Behaviors for Automated Gameplay. IEEE Transactions on Games 15, 3 (2023), 469--482. https://doi.org/10.1109/TG.2023.3241864
[12]
Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology. Vol. 52. Elsevier, 139--183.
[13]
Austin Z. Henley and Scott D. Fleming. 2018. CodeDeviant: Helping Programmers Detect Edits That Accidentally Alter Program Behavior. In 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). 65--73. https://doi.org/10.1109/VLHCC.2018.8506567
[14]
Peiling Jiang, Fuling Sun, and Haijun Xia. 2023. Log-It: Supporting Programming with Interactive, Contextual, Structured, and Visual Logs. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 594, 16 pages. https://doi.org/10.1145/3544548.3581403
[15]
Majeed Kazemitabaar, Xinying Hou, Austin Henley, Barbara J. Ericson, David Weintrop, and Tovi Grossman. 2023. How Novices Use LLM-Based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environment. arXiv:2309.14049 [cs.HC]
[16]
Yoonji Kim, Youngkyung Choi, Daye Kang, Minkyeong Lee, Tek-Jin Nam, and Andrea Bianchi. 2020. HeyTeddy: Conversational Test-Driven Development for Physical Computing. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 4, Article 139 (sep 2020), 21 pages. https://doi.org/10.1145/3369838
[17]
Yoonji Kim, Youngkyung Choi, Hyein Lee, Geehyuk Lee, and Andrea Bianchi. 2019. VirtualComponent: A Mixed-Reality Tool for Designing and Tuning Breadboarded Circuits. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--13. https://doi.org/10.1145/3290605.3300407
[18]
Yoonji Kim, Hyein Lee, Ramkrishna Prasad, Seungwoo Je, Youngkyung Choi, Daniel Ashbrook, Ian Oakley, and Andrea Bianchi. 2020. SchemaBoard: Supporting Correct Assembly of Schematic Circuits using Dynamic In-Situ Visualization. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST '20). Association for Computing Machinery, New York, NY, USA, 987--998. https://doi.org/10.1145/3379337.3415887
[19]
Amy J. Ko and Brad A. Myers. 2004. Designing the Whyline: A Debugging Interface for Asking Questions about Program Behavior. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vienna, Austria) (CHI '04). Association for Computing Machinery, New York, NY, USA, 151--158. https://doi.org/10.1145/985692.985712
[20]
Amy J. Ko and Brad A. Myers. 2008. Debugging Reinvented: Asking and Answering Why and Why Not Questions about Program Behavior. In Proceedings of the 30th International Conference on Software Engineering (Leipzig, Germany) (ICSE '08). Association for Computing Machinery, New York, NY, USA, 301--310. https://doi.org/10.1145/1368088.1368130
[21]
Amy J. Ko, Brad A. Myers, and Htet Htet Aung. 2004. Six Learning Barriers in End-User Programming Systems. In 2004 IEEE Symposium on Visual Languages - Human Centric Computing. IEEE, USA, 199--206. https://doi.org/10.1109/VLHCC.2004.47
[22]
Amy J. Ko, Brad A. Myers, Michael J. Coblenz, and Htet Htet Aung. 2006. An Exploratory Study of How Developers Seek, Relate, and Collect Relevant Information during Software Maintenance Tasks. IEEE Transactions on Software Engineering 32, 12 (2006), 971--987. https://doi.org/10.1109/TSE.2006.116
[23]
Thomas D. LaToza and Brad A. Myers. 2010. Developers Ask Reachability Questions. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1 (Cape Town, South Africa) (ICSE '10). Association for Computing Machinery, New York, NY, USA, 185--194. https://doi.org/10.1145/1806799.1806829
[24]
Woojin Lee, Ramkrishna Prasad, Seungwoo Je, Yoonji Kim, Ian Oakley, Daniel Ashbrook, and Andrea Bianchi. 2021. VirtualWire: Supporting Rapid Prototyping with Instant Reconfigurations of Wires in Breadboarded Circuits. In Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction (Salzburg, Austria) (TEI '21). Association for Computing Machinery, New York, NY, USA, 1--12. https://doi.org/10.1145/3430524.3440623
[25]
James R Lewis. 2018. The system usability scale: past, present, and future. International Journal of Human-Computer Interaction 34, 7 (2018), 577--590.
[26]
Tom Lieber, Joel R. Brandt, and Rob C. Miller. 2014. Addressing Misconceptions about Code with Always-on Programming Visualizations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI '14). Association for Computing Machinery, New York, NY, USA, 2481--2490. https://doi.org/10.1145/2556288.2557409
[27]
Richard Lin, Rohit Ramesh, Nikhil Jain, Josephine Koe, Ryan Nuqui, Prabal Dutta, and Bjoern Hartmann. 2021. Weaving Schematics and Code: Interactive Visual Editing for Hardware Description Languages. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST '21). Association for Computing Machinery, New York, NY, USA, 1039--1049. https://doi.org/10.1145/3472749.3474804
[28]
Michael Xieyang Liu, Advait Sarkar, Carina Negreanu, Benjamin Zorn, Jack Williams, Neil Toronto, and Andrew D. Gordon. 2023. "What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 598, 31 pages. https://doi.org/10.1145/3544548.3580817
[29]
Jo-Yu Lo, Da-Yuan Huang, Tzu-Sheng Kuo, Chen-Kuo Sun, Jun Gong, Teddy Seyed, Xing-Dong Yang, and Bing-Yu Chen. 2019. AutoFritz: Autocomplete for Prototyping Virtual Breadboard Circuits. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--13. https://doi.org/10.1145/3290605.3300633
[30]
Cong Lu, Raluca Georgescu, and Johan Verwey. 2022. Go-Explore Complex 3D Game Environments for Automated Reachability Testing. IEEE Transactions on Games (2022), 1--6. https://doi.org/10.1109/TG.2022.3228401
[31]
Will McGrath, Daniel Drew, Jeremy Warner, Majeed Kazemitabaar, Mitchell Karchemsky, David Mellis, and Björn Hartmann. 2017. Bifröst: Visualizing and Checking Behavior of Embedded Systems across Hardware and Software. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST '17). Association for Computing Machinery, New York, NY, USA, 299--310. https://doi.org/10.1145/3126594.3126658
[32]
Yoichi Ochiai. 2014. Visible Breadboard: System for Dynamic, Programmable, and Tangible Circuit Prototyping with Visible Electricity. In Proceedings of the 6th International Conference on Virtual, Augmented and Mixed Reality. Applications of Virtual and Augmented Reality - Volume 8526. Springer-Verlag, Berlin, Heidelberg, 73--84. https://doi.org/10.1007/978-3-319-07464-1_7
[33]
Dan O'Sullivan and Tom Igoe. 2004. Physical computing: sensing and controlling the physical world with computers. Course Technology Press.
[34]
Fabio Paternò, Antonio Giovanni Schiavone, and Antonio Conti. 2017. Customizable automatic detection of bad usability smells in mobile accessed web applications. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (Vienna, Austria) (MobileHCI '17). Association for Computing Machinery, New York, NY, USA, Article 42, 11 pages. https://doi.org/10.1145/3098279.3098558
[35]
David Piorkowski, Austin Z. Henley, Tahmid Nabi, Scott D. Fleming, Christopher Scaffidi, and Margaret Burnett. 2016. Foraging and Navigations, Fundamentally: Developers' Predictions of Value and Cost. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (Seattle, WA, USA) (FSE 2016). Association for Computing Machinery, New York, NY, USA, 97--108. https://doi.org/10.1145/2950290.2950302
[36]
David J. Piorkowski, Scott D. Fleming, Irwin Kwan, Margaret M. Burnett, Christopher Scaffidi, Rachel K.E. Bellamy, and Joshua Jordahl. 2013. The Whats and Hows of Programmers' Foraging Diets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI '13). Association for Computing Machinery, New York, NY, USA, 3063--3072. https://doi.org/10.1145/2470654.2466418
[37]
Mareen Przybylla and Ralf Romeike. 2017. The Nature of Physical Computing in Schools: Findings from Three Years of Practical Experience. In Proceedings of the 17th Koli Calling International Conference on Computing Education Research (Koli, Finland) (Koli Calling '17). Association for Computing Machinery, New York, NY, USA, 98--107. https://doi.org/10.1145/3141880.3141889
[38]
Patrick Rein, Stefan Ramson, Jens Lincke, Robert Hirschfeld, and Tobias Pape. 2018. Exploratory and live, programming and coding: a literature study comparing perspectives on liveness. arXiv preprint arXiv:1807.08578 (2018).
[39]
Joel Sadler, Kevin Durfee, Lauren Shluzas, and Paulo Blikstein. 2015. Bloctopus: A Novice Modular Sensor System for Playful Prototyping. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (Stanford, California, USA) (TEI '15). Association for Computing Machinery, New York, NY, USA, 347--354. https://doi.org/10.1145/2677199.2680581
[40]
Konstantin Schekotihin, Birgit Hofer, Franz Wotawa, and Dietmar Jannach. 2021. AI-based Spreadsheet Debugging. 371--399. https://doi.org/10.1142/9789811239922_0013
[41]
Peter Seibel. 2009. Coders at Work (1st ed.). Apress, USA.
[42]
Jonathan Sillito, Gail C. Murphy, and Kris De Volder. 2008. Asking and Answering Questions during a Programming Change Task. IEEE Transactions on Software Engineering 34, 4 (2008), 434--451. https://doi.org/10.1109/TSE.2008.26
[43]
Evan Strasnick, Maneesh Agrawala, and Sean Follmer. 2017. Scanalog: Interactive Design and Debugging of Analog Circuits with Programmable Hardware. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST '17). Association for Computing Machinery, New York, NY, USA, 321--330. https://doi.org/10.1145/3126594.3126618
[44]
Evan Strasnick, Maneesh Agrawala, and Sean Follmer. 2021. Coupling Simulation and Hardware for Interactive Circuit Debugging. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 667, 15 pages. https://doi.org/10.1145/3411764.3445422
[45]
Evan Strasnick, Maneesh Agrawala, and Sean Follmer. 2021. Coupling Simulation and Hardware for Interactive Circuit Debugging. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, 1--15. https://doi.org/10.1145/3411764.3445422
[46]
Evan Strasnick, Sean Follmer, and Maneesh Agrawala. 2019. Pinpoint: A PCB Debugging Pipeline Using Interruptible Routing and Instrumentation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--11. https://doi.org/10.1145/3290605.3300278
[47]
Bret Victor. 2014. Humane Representation of Thought: A Trail Map for the 21st Century. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (Honolulu, Hawaii, USA) (UIST '14). Association for Computing Machinery, New York, NY, USA, 699. https://doi.org/10.1145/2642918.2642920
[48]
Chiuan Wang, Hsuan-Ming Yeh, Bryan Wang, Te-Yen Wu, Hsin-Ruey Tsai, Rong-Hao Liang, Yi-Ping Hung, and Mike Y. Chen. 2016. CircuitStack: Supporting Rapid Prototyping and Evolution of Electronic Circuits. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (Tokyo, Japan) (UIST '16). Association for Computing Machinery, New York, NY, USA, 687--695. https://doi.org/10.1145/2984511.2984527
[49]
Jeremy Warner, Ben Lafreniere, George Fitzmaurice, and Tovi Grossman. 2018. ElectroTutor: Test-Driven Physical Computing Tutorials. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (Berlin, Germany) (UIST '18). Association for Computing Machinery, New York, NY, USA, 435--446. https://doi.org/10.1145/3242587.3242591
[50]
Stephan Wensveen and Ben Matthews. 2014. Prototypes and prototyping in design research. In The routledge companion to design research. Routledge, 262--276.
[51]
Te-Yen Wu, Jun Gong, Teddy Seyed, and Xing-Dong Yang. 2019. Proxino: Enabling Prototyping of Virtual Circuits with Physical Proxies. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST '19). Association for Computing Machinery, New York, NY, USA, 121--132. https://doi.org/10.1145/3332165.3347938
[52]
Te-Yen Wu, Hao-Ping Shen, Yu-Chian Wu, Yu-An Chen, Pin-Sung Ku, Ming-Wei Hsu, Jun-You Liu, Yu-Chih Lin, and Mike Y. Chen. 2017. CurrentViz: Sensing and Visualizing Electric Current Flows of Breadboarded Circuits. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST '17). Association for Computing Machinery, New York, NY, USA, 343--349. https://doi.org/10.1145/3126594.3126646
[53]
Te-Yen Wu, Bryan Wang, Jiun-Yu Lee, Hao-Ping Shen, Yu-Chian Wu, Yu-An Chen, Pin-Sung Ku, Ming-Wei Hsu, Yu-Chih Lin, and Mike Y. Chen. 2017. CircuitSense: Automatic Sensing of Physical Circuits and Generation of Virtual Circuits to Support Software Tools. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST '17). Association for Computing Machinery, New York, NY, USA, 311--319. https://doi.org/10.1145/3126594.3126634
Index Terms
- Inline Visualization and Manipulation of Real-Time Hardware Log for Supporting Debugging of Embedded Programs
Recommendations
Replay Debugging of Real-Time Vxworks Applications
COMPSAC '15: Proceedings of the 2015 IEEE 39th Annual Computer Software and Applications Conference - Volume 03Debugging multi-task real-time VxWorks applications is tedious and time-consuming for developers. The non-determinism within the application execution makes the developers hard to reproduce a failure.1As a result, the developers cannot perform cyclic ...
A hardware time manager implementation for the Xenomai real-time kernel of embedded Linux
2nd Workshop on Embed With Linux (EWiLi 2012)Nowadays, the use of embedded operating systems in different embedded projects is subject to a tremendous growth. Embedded Linux is becoming one of those most popular EOSs due to its modularity, efficiency, reliability, and cost. One way to make it hard ...
Stepwise Debugging for Hardware Accelerators
ASPLOS 2023: Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2High-level programming models for hardware design let domain experts quickly produce specialized accelerators. However, tools for debugging these accelerators remain tied to low-level hardware description languages (HDLs). High-level descriptions ...
Comments
Information & Contributors
Information
Published In
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 17 June 2024
Accepted: 01 December 2023
Received: 01 October 2023
Published in PACMHCI Volume 8, Issue EICS
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed
Funding Sources
- National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT)
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 135Total Downloads
- Downloads (Last 12 months)135
- Downloads (Last 6 weeks)26
Reflects downloads up to 15 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in