research-article

Open access

Math Augmentation: How Authors Enhance the Readability of Formulas using Novel Visual Design Practices

Authors:

Marti A. HearstAuthors Info & Claims

CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

Article No.: 491, Pages 1 - 18

https://doi.org/10.1145/3491102.3501932

Published: 29 April 2022 Publication History

All formats PDF

Abstract

With the increasing growth and impact of machine learning and other math-intensive fields, it is more important than ever to broaden access to mathematical notation. Can new visual and interactive displays help a wider readership successfully engage with notation? This paper provides the first detailed qualitative analysis of math augmentation—the practice of embellishing notation with novel visual design patterns to improve its readability. We present two qualitative studies of the practice of math augmentation. First is an analysis of 1.1k augmentations to 281 formulas in 47 blogs, textbooks, and other documents containing mathematical expressions. Second is an interview study with 12 authors who had previously designed custom math augmentations (“maugs”). This paper contributes a comprehensive inventory of the kinds of maugs that appear in math documents, and a detailed account of how authors’ tools ought to be redesigned to support efficient creation of math augmentations. These studies open a critical new design space for HCI researchers and interface designers.

Supplemental Material

MP4 File

Talk Video

Transcript for: Talk Video

MP4 File

Video Preview

Download
4.48 MB

ZIP File

Supplemental Materials

Download
11.02 MB

References

[1]

Thomasenia Lott Adams. 2003. Reading mathematics: More than words can say. The Reading Teacher 56, 8 (2003).

[2]

Lara Alcock and Nicola Wilkinson. 2011. e-Proofs: Design of a Resource to Support Proof Comprehension in Mathematics. Educational Designer 1, 4 (2011).

[3]

Andrew M.H. Alexander. Exponential Improvements. http://www.andrusia.com/math/complex-numbers/eulers-identity.pdf Last accessed December 27, 2021.

[4]

Sriram Karthik Badam, Zhicheng Liu, and Niklas Elmqvist. 2019. Elastic Documents: Coupling Text and Tables through Contextual Visualizations for Enhanced Document Reading. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2019).

Digital Library

[5]

Sergio Basurco. Isaac Hayes Wallpaper Generator - Volumetric light scattering, 1 of 2. https://chuckleplant.github.io/2017/05/28/light-shafts.html Last accessed December 27, 2021.

[6]

Tali Benyon. Build your own Tensorflow. https://taliesin.ai/projects/edu/indaba-2019/ Last accessed December 27, 2021.

[7]

Chris Bryan, Kwan-Liu Ma, and Jonathan Woodring. 2017. Temporal Summary Images: An Approach to Narrative Visualization via Interactive Annotation Generation and Placement. IEEE Transactions on Visualization and Computer Graphics 23, 1 (2017).

Digital Library

[8]

Carrie J. Cai and Philip J. Guo. 2019. Software Developers Learning Machine Learning: Motivations, Hurdles, and Desires. In Proceedings of the Symposium on Visual Languages and Human-Centric Computing.

[9]

David P. Carlisle. Thecolorpackage, CTAN-Archive.

[10]

Rowan Cockett, Lindsey Heagy, and Doug Oldenburg. 2016. Pixels and their neighbors: Finite volume. The Leading Edge 35, 8 (2016).

[11]

Matthew Conlen and Jeffrey Heer. 2018. Idyll: A Markup Language for Authoring and Publishing Interactive Articles on the Web. In Proceedings of the Symposium on User Interface Software and Technology.

Digital Library

[12]

Juliet Corbin and Anselm Strauss. 2008. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory (3rd ed.). SAGE Publications.

[13]

Yancarlos Diaz, Gavin Nishizawa, Behrooz Mansouri, Kenny Davila, and Richard Zanibbi. 2021. The MathDeck Formula Editor: Interactive Formula Entry Combining LaTeX, Structure Editing, and Search. In Proceedings of the CHI Conference on Human Factors in Computing Systems.

Digital Library

[14]

Pierre Dragicevic, Yvonne Jansen, Abhraneel Sarma, Matthew Kay, and Fanny Chevalier. 2019. Increasing the Transparency of Research Papers with Explorable Multiverse Analyses. In Proceedings of the CHI Conference on Human Factors in Computing Systems.

Digital Library

[15]

Anton N. Dragunov and Jonathan L. Herlocker. 2003. Designing intelligent and dynamic interfaces for communicating mathematics. In Proceedings of the International Conference on Intelligent User Interfaces.

[16]

Emily Eisenberg and Sophie Alpert. KaTeX. https://katex.org Last accessed September 16, 2020.

[17]

Tong Ge, Yue Zhao, Bongshin Lee, Donghao Ren, Baoquan Chen, and Yunhai Wang. 2020. Canis: A High‐Level Language for Data‐Driven Chart Animations. Computer Graphics Forum 39 (2020).

[18]

Gabriel Goh. 2017. Why momentum really works. Distill 2, 4 (2017). http://distill.pub/2017/momentum

[19]

Sam Greydanus and Chris Olah. 2019. The Paths Perspective on Value Learning. In Distill. https://distill.pub/2019/paths-perspective-on-value-learning

[20]

Bobby He, Balaji Lakshminarayanan, and Yee Whye Teh. Bayesian Deep Ensembles via the Neural Tangent Kernel, NeurIPs Poster Presentation. https://nips.cc/virtual/2020/protected/poster_0b1ec366924b26fc98fa7b71a9c249cf.html Last accessed December 30, 2021.

[21]

Andrew Head, Codanda Appachu, Marti A. Hearst, and Björn Hartmann. 2015. Tutorons: Generating context-relevant, on-demand explanations and demonstrations of online code. In Proceedings of the Symposium on Visual Languages and Human-Centric Computing.

[22]

Andrew Head, Kyle Lo, Dongyeop Kang, Raymond Fok, Sam Skjonsberg, Daniel S. Weld, and Marti A. Hearst. 2021. Augmenting Scientific Papers with Just-in-Time, Position-Sensitive Definitions of Terms and Symbols. In Proceedings of the CHI Conference on Human Factors in Computing Systems.

Digital Library

[23]

Fred Hohman, Matthew Conlen, Jeffrey Heer, and Duen Horng Chau. 2020. Communicating with Interactive Articles. distill.pub.

[24]

Fred Hohman, Andrew Head, Rich Caruana, Robert DeLine, and Steven M. Drucker. 2019. Gamut: A design probe to understand how data scientists understand machine learning models. In Proceedings of the CHI Conference on Human Factors in Computing Systems.

Digital Library

[25]

Fred Hohman and other contributors. 2020. Awesome Mathematical Notation Design. https://github.com/fredhohman/awesome-mathematical-notation-design

[26]

Ole Holsti. 1969. Content Analysis for the Social Sciences and Humanities. Addison-Wesley.

[27]

Jessica Hullman and Nicholas Diakopoulos. 2011. Visualization Rhetoric: Framing Effects in Narrative Visualization. IEEE Transactions on Visualization and Computer Graphics 17(2011).

[28]

Jessical Hullman, Nicholas Diakopoulos, and Eytan Adar. 2013. Contextifier: automatic generation of annotated stock visualizations. In Proceedings of the CHI Conference on Human Factors in Computing Systems.

Digital Library

[29]

Jessica Hullman, Yea-Seul Kim, Frances Nguyen, Lauren Speers, and Maneesh Agrawala. 2018. Improving Comprehension of Measurements Using Concrete Re-expression Strategies. In Proceedings of the CHI Conference on Human Factors in Computing Systems.

Digital Library

[30]

Ankit Jain, Isaac Liu, Ankur Sarda, and Piero Molino. Food Discovery with Uber Eats: Using Graph Learning to Power Recommendations. https://eng.uber.com/uber-eats-graph-learning/ Last accessed December 30, 2021.

[31]

Anthony R. Jansen, Kim Marriott, and Greg Yelland. 1999. Perceiving structure in mathematical expressions. In Proceedings of the Conference of the Cognitive Science Society.

[32]

Seokbin Kang, Ekta Shokeen, Virginia L. Byrne, Leyla Norooz, Elizabeth M. Bonsignore, Caro Williams-Pierce, and Jon Froehlich. 2020. ARMath: Augmenting Everyday Life with Math Learning. In Proceedings of the CHI Conference on Human Factors in Computing Systems.

Digital Library

[33]

Dae Hyun Kim, Enamul Hoque, Juho Kim, and Maneesh Agrawala. 2018. Facilitating Document Reading by Linking Text and Tables. In Proceedings of the Symposium on User Interface Software and Technology.

Digital Library

[34]

Hwiyeon Kim, Juyoung Oh, Yunha Han, Sungahn Ko, Matthew Brehmer, and Bum Chul Kwon. 2019. Thumbnails for Data Stories: A Survey of Current Practices. In Proceedings of the IEEE Visualization Conference.

[35]

Donald Ervin Knuth. 1986. Computers & Typesetting, Vol. A: The TeX book. Addison-Wesley Longman Publishing Co., Inc.

[36]

Andrea Kohlhase, Michael Kohlhase, and Taweechai Ouypornkochagorn. 2018. Discourse Phenomena in Mathematical Documents. In Intelligent Computer Mathematics, Florian Rabe, William M. Farmer, Grant O. Passmore, and Abdou Youssef (Eds.).

[37]

Ha-kyung Kong, Wenjie Zhu, Zhicheng Liu, and Karrie Karahalios. 2019. Understanding Visual Cues in Visualizations Accompanied by Audio Narrations. In Proceedings of the CHI Conference on Human Factors in Computing Systems.

[38]

Nicholas Kong, Marti A. Hearst, and Maneesh Agrawala. 2014. Extracting references between text and charts via crowdsourcing. In Proceedings of the CHI Conference on Human Factors in Computing Systems.

Digital Library

[39]

David Landy and Robert L. Goldstone. 2007. Formal notations are diagrams: Evidence from a production task. Memory & Cognition 35(2007).

[40]

Shahid Latif, Zheng Zhou, Yoon Kim, Fabian Beck, and Nam Wook Kim. 2021. Kori: Interactive Synthesis of Text and Charts in Data Documents. In Proceedings of the IEEE Visualization Conference.

[41]

Junhua Lu, Wei Chen, Hui Ye, Jie Wang, Honghui Mei, Yuhui Gu, Yingcai Wu, Xiaolong Luke Zhang, and Kwan-Liu Ma. 2021. Automatic Generation of Unit Visualization-based Scrollytelling for Impromptu Data Facts Delivery. In Proceedings of the IEEE Pacific Visualization Symposium (PacificVis).

[42]

Lars Madsen, Will Robertson, and Joseph Wright. The mathtools package, CTAN-Archive, ctan.org.

[43]

Chenlin Meng, Lantao Yu, Yang Song, Jiaming Song, and Stefano Ermon. Autoregressive Score Matching, NeurIPs Poster Presentation. https://nips.cc/virtual/2020/protected/poster_4a4526b1ec301744aba9526d78fcb2a6.html Last accessed December 30, 2021.

[44]

Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt. Topological Autoencoders, ICML Poster Presentation. https://icml.cc/virtual/2020/poster/5851 Last accessed December 30, 2021.

[45]

Jousef Murad. Derivation of the Navier-Stokes Equations. https://www.youtube.com/watch?v=zWdnf3Uh1RE Last accessed December 30, 2021.

[46]

Magnus Österholm. 2006. Characterizing Reading Comprehension of Mathematical Texts. Educational Studies in Mathematics 63 (2006).

[47]

Ari Pakman, Yueqi Wang, Catalin Mitelut, and JinHyung Lee. Neural Clustering Processes, ICML Poster Presentation (slide 27). https://icml.cc/virtual/2020/poster/6425 Last accessed December 30, 2021.

[48]

Victor Powell. Image Kernels Explained Visually. https://setosa.io/ev/image-kernels/ Last accessed December 30, 2021.

[49]

Edward Raff. 2020. Inside Deep Learning: Math, Algorithms, Models. Manning.

[50]

Donghao Ren, M. Brehmer, Bongshin Lee, Tobias Höllerer, and E. Choe. 2017. ChartAccent: Annotation for data-driven storytelling. In Proceedings of the IEEE Pacific Visualization Symposium (PacificVis).

[51]

Nicholas Rougeux. Byrne’s Euclid. https://www.c82.net/euclid/book1 Last accessed December 30, 2021.

[52]

Grant Sanderson. Manim. https://3b1b.github.io/manim/

[53]

Nazmus Saquib, Rubaiat Habib Kazi, Li-yi Wei, Gloria Mark, and Deb Roy. 2021. Constructing Embodied Algebra by Sketching. In Proceedings of the CHI Conference on Human Factors in Computing Systems.

Digital Library

[54]

Alper Sarikaya, M. Correll, L. Bartram, Melanie K. Tory, and Danyel Fisher. 2019. What Do We Talk About When We Talk About Dashboards?IEEE Transactions on Visualization and Computer Graphics 25(2019).

[55]

Arvind Satyanarayan and Jeffrey Heer. 2014. Authoring Narrative Visualizations with Ellipsis. Computer Graphics Forum 33 (2014).

[56]

Arvind Satyanarayan, Dominik Moritz, Kanit Wongsuphasawat, and Jeffrey Heer. 2017. Vega-Lite: A Grammar of Interactive Graphics. In IEEE Transactions on Visualization and Computer Graphics, Vol. 23.

[57]

Mary D. Shepherd and Carla V. D. Sande. 2014. Reading mathematics for understanding—From novice to expert. The Journal of Mathematical Behavior 35 (2014).

[58]

Arjun Srinivasan, Steven Drucker, Alex Endert, and John Stasko. 2019. Augmenting Visualizations with Interactive Data Facts to Facilitate Interpretation and Communication. IEEE Transactions on Visualization and Computer Graphics 25(2019).

Digital Library

[59]

Edward Tufte. 1985. The visual display of quantitative information. Graphics Press.

Digital Library

[60]

Bret Victor. Tangle: a JavaScript library for reactive documents. http://worrydream.com/Tangle/ Last accessed August 15, 2021.

[61]

Theresa E. Wege, Sophie Batchelor, Matthew Inglis, Honali Mistry, and Dirk Schlimm. 2020. Iconicity in Mathematical Notation: Commutativity and Symmetry. Journal of Numerical Cognition 6 (2020).

[62]

Katherine Ye, Wode Ni, Max Krieger, Dor Ma’ayan, Jenna Wise, Jonathan Aldrich, Joshua Sunshine, and Keenan Crane. 2020. Penrose: from mathematical notation to beautiful diagrams. ACM Transactions on Graphics (TOG) 39 (2020).

Digital Library

[63]

Hsin I. Yung and Fred Paas. 2015. Effects of Computer-Based Visual Representation on Mathematics Learning and Cognitive Load. J. Educ. Technol. Soc. 18 (2015).

[64]

Robert C. Zeleznik, Andrew Bragdon, Ferdi Adeputra, and Hsu-Sheng Ko. 2010. Hands-on math: a page-based multi-touch and pen desktop for technical work and problem solving. In Proceedings of the Symposium on User Interface Software and Technology.

Digital Library

Cited By

Nassif MRobillard M(2024)Non-Linear Software Documentation with Interactive Code ExamplesACM Transactions on Software Engineering and Methodology10.1145/370297634:2(1-32)Online publication date: 5-Nov-2024
https://dl.acm.org/doi/10.1145/3702976
Goree SAppleby GCrandall DSu N(2024)Attention is All They Need: Exploring the Media Archaeology of the Computer Vision Research PaperProceedings of the ACM on Human-Computer Interaction10.1145/36869558:CSCW2(1-25)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686955
Lo KChang JHead ABragg JZhang ATrier CAnastasiades CAugust TAuthur RBragg DBransom ECachola ICandra SChandrasekhar YChen YCheng EChou YDowney DEvans RFok RHu FHuff RKang DKim TKinney RKittur AKang HKlevak EKuehl BLangan MLatzke MLochner JMacMillan KMarsh EMurray TNaik ANguyen NPalani SPark SPaulic CRachatasumrit NRao SSayre PShen ZSiangliulue PSoldaini LTran Hvan Zuylen MWang LWilhelm CWu CYang JZamarron AHearst MWeld D(2024)The Semantic Reader ProjectCommunications of the ACM10.1145/365909667:10(50-61)Online publication date: 26-Sep-2024
https://dl.acm.org/doi/10.1145/3659096
Show More Cited By

Index Terms

Math Augmentation: How Authors Enhance the Readability of Formulas using Novel Visual Design Practices
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interactive systems and tools

Recommendations

Visual Link Routing in Immersive Visualisations
ISS '19: Proceedings of the 2019 ACM International Conference on Interactive Surfaces and Spaces

In immersive display environments, such as virtual or augmented reality, we can make explicit the connections between data points in visualisations and their context in the world, or in other visualisations. This paper considers the requirements and ...
Situated practices of looking: visual practice in an online world
CSCW '08: Proceedings of the 2008 ACM conference on Computer supported cooperative work

Graphical virtual worlds are increasingly significant sites of collaborative interaction. Many argue that the simulation of the everyday environment makes them particularly effective for collaboration. Based on a study of visual practice in Second Life, ...
Drawing Connections: Designing Situated Links for Immersive Maps
MHCI

We explore the design of situated visual links in outdoor augmented reality (AR) for connecting miniature buildings on a virtual map to their real-world counterparts. We first distill design criteria from prior work, then conduct two user studies to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

April 2022

10459 pages

ISBN:9781450391573

DOI:10.1145/3491102

Editors:
Simone Barbosa
PUC-Rio, Brazil
,
Cliff Lampe
University of Michigan, USA
,
Caroline Appert
Université Paris-Saclay, France
,
David A. Shamma
Toyota Research Institute, USA
,
Steven Drucker
Microsoft Research, USA
,
Julie Williamson
University of Glasgow, UK
,
Koji Yatani
University of Tokyo, Japan

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike International 4.0 License.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 April 2022

Check for updates

Badges

Best Paper

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CHI '22

Sponsor:

SIGCHI

CHI '22: CHI Conference on Human Factors in Computing Systems

April 29 - May 5, 2022

LA, New Orleans, USA

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
16,271
Total Downloads

Downloads (Last 12 months)11,011
Downloads (Last 6 weeks)463

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Nassif MRobillard M(2024)Non-Linear Software Documentation with Interactive Code ExamplesACM Transactions on Software Engineering and Methodology10.1145/370297634:2(1-32)Online publication date: 5-Nov-2024
https://dl.acm.org/doi/10.1145/3702976
Goree SAppleby GCrandall DSu N(2024)Attention is All They Need: Exploring the Media Archaeology of the Computer Vision Research PaperProceedings of the ACM on Human-Computer Interaction10.1145/36869558:CSCW2(1-25)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686955
Lo KChang JHead ABragg JZhang ATrier CAnastasiades CAugust TAuthur RBragg DBransom ECachola ICandra SChandrasekhar YChen YCheng EChou YDowney DEvans RFok RHu FHuff RKang DKim TKinney RKittur AKang HKlevak EKuehl BLangan MLatzke MLochner JMacMillan KMarsh EMurray TNaik ANguyen NPalani SPark SPaulic CRachatasumrit NRao SSayre PShen ZSiangliulue PSoldaini LTran Hvan Zuylen MWang LWilhelm CWu CYang JZamarron AHearst MWeld D(2024)The Semantic Reader ProjectCommunications of the ACM10.1145/365909667:10(50-61)Online publication date: 26-Sep-2024
https://dl.acm.org/doi/10.1145/3659096
Gunturu AWen YZhang NThundathil JKazi RSuzuki R(2024)Augmented Physics: Creating Interactive and Embedded Physics Simulations from Static Textbook DiagramsProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676392(1-12)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676392
Hayatpur DHempel BChen KDuan WGuo PXia H(2024)Taking ASCII Drawings Seriously: How Programmers Diagram CodeProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642683(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642683
Liu MWu TChen TLi FKittur AMyers B(2024)Selenite: Scaffolding Online Sensemaking with Comprehensive Overviews Elicited from Large Language ModelsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642149(1-26)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642149
Lee MChen B(2024)MaugVLink: Augmenting Mathematical Formulas with Visual Links2024 IEEE 17th Pacific Visualization Conference (PacificVis)10.1109/PacificVis60374.2024.00048(337-342)Online publication date: 23-Apr-2024
https://doi.org/10.1109/PacificVis60374.2024.00048
Miah MKong J(2024)Augmented Reality and Cross-Device Interaction for Seamless Integration of Physical and Digital Scientific PapersInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2388372(1-18)Online publication date: 22-Aug-2024
https://doi.org/10.1080/10447318.2024.2388372
Beaudouin-Lafon ME JXia H(2023)Color Field: Developing Professional Vision by Visualizing the Effects of Color FiltersProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606828(1-16)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606828
Chulpongsatorn NLunding MSoni NSuzuki R(2023)Augmented Math: Authoring AR-Based Explorable Explanations by Augmenting Static Math TextbooksProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606827(1-16)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606827
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten