-
Using Large Language Models to Accelerate Communication for Users with Severe Motor Impairments
Authors:
Shanqing Cai,
Subhashini Venugopalan,
Katie Seaver,
Xiang Xiao,
Katrin Tomanek,
Sri Jalasutram,
Meredith Ringel Morris,
Shaun Kane,
Ajit Narayanan,
Robert L. MacDonald,
Emily Kornman,
Daniel Vance,
Blair Casey,
Steve M. Gleason,
Philip Q. Nelson,
Michael P. Brenner
Abstract:
Finding ways to accelerate text input for individuals with profound motor impairments has been a long-standing area of research. Closing the speed gap for augmentative and alternative communication (AAC) devices such as eye-tracking keyboards is important for improving the quality of life for such individuals. Recent advances in neural networks of natural language pose new opportunities for re-thi…
▽ More
Finding ways to accelerate text input for individuals with profound motor impairments has been a long-standing area of research. Closing the speed gap for augmentative and alternative communication (AAC) devices such as eye-tracking keyboards is important for improving the quality of life for such individuals. Recent advances in neural networks of natural language pose new opportunities for re-thinking strategies and user interfaces for enhanced text-entry for AAC users. In this paper, we present SpeakFaster, consisting of large language models (LLMs) and a co-designed user interface for text entry in a highly-abbreviated form, allowing saving 57% more motor actions than traditional predictive keyboards in offline simulation. A pilot study with 19 non-AAC participants typing on a mobile device by hand demonstrated gains in motor savings in line with the offline simulation, while introducing relatively small effects on overall typing speed. Lab and field testing on two eye-gaze typing users with amyotrophic lateral sclerosis (ALS) demonstrated text-entry rates 29-60% faster than traditional baselines, due to significant saving of expensive keystrokes achieved through phrase and word predictions from context-aware LLMs. These findings provide a strong foundation for further exploration of substantially-accelerated text communication for motor-impaired users and demonstrate a direction for applying LLMs to text-based user interfaces.
△ Less
Submitted 3 December, 2023;
originally announced December 2023.
-
LaMPost: Design and Evaluation of an AI-assisted Email Writing Prototype for Adults with Dyslexia
Authors:
Steven M. Goodman,
Erin Buehler,
Patrick Clary,
Andy Coenen,
Aaron Donsbach,
Tiffanie N. Horne,
Michal Lahav,
Robert Macdonald,
Rain Breaw Michaels,
Ajit Narayanan,
Mahima Pushkarna,
Joel Riley,
Alex Santana,
Lei Shi,
Rachel Sweeney,
Phil Weaver,
Ann Yuan,
Meredith Ringel Morris
Abstract:
Prior work has explored the writing challenges experienced by people with dyslexia, and the potential for new spelling, grammar, and word retrieval technologies to address these challenges. However, the capabilities for natural language generation demonstrated by the latest class of large language models (LLMs) highlight an opportunity to explore new forms of human-AI writing support tools. In thi…
▽ More
Prior work has explored the writing challenges experienced by people with dyslexia, and the potential for new spelling, grammar, and word retrieval technologies to address these challenges. However, the capabilities for natural language generation demonstrated by the latest class of large language models (LLMs) highlight an opportunity to explore new forms of human-AI writing support tools. In this paper, we introduce LaMPost, a prototype email-writing interface that explores the potential for LLMs to power writing support tools that address the varied needs of people with dyslexia. LaMPost draws from our understanding of these needs and introduces novel AI-powered features for email-writing, including: outlining main ideas, generating a subject line, suggesting changes, rewriting a selection. We evaluated LaMPost with 19 adults with dyslexia, identifying many promising routes for further exploration (including the popularity of the "rewrite" and "subject line" features), but also finding that the current generation of LLMs may not surpass the accuracy and quality thresholds required to meet the needs of writers with dyslexia. Surprisingly, we found that participants' awareness of the AI had no effect on their perception of the system, nor on their feelings of autonomy, expression, and self-efficacy when writing emails. Our findings yield further insight into the benefits and drawbacks of using LLMs as writing support for adults with dyslexia and provide a foundation to build upon in future research.
△ Less
Submitted 5 July, 2022;
originally announced July 2022.
-
LAMP: Learning a Motion Policy to Repeatedly Navigate in an Uncertain Environment
Authors:
Florence Tsang,
Tristan Walker,
Ryan A. MacDonald,
Armin Sadeghi,
Stephen L. Smith
Abstract:
Mobile robots are often tasked with repeatedly navigating through an environment whose traversability changes over time. These changes may exhibit some hidden structure, which can be learned. Many studies consider reactive algorithms for online planning, however, these algorithms do not take advantage of the past executions of the navigation task for future tasks. In this paper, we formalize the p…
▽ More
Mobile robots are often tasked with repeatedly navigating through an environment whose traversability changes over time. These changes may exhibit some hidden structure, which can be learned. Many studies consider reactive algorithms for online planning, however, these algorithms do not take advantage of the past executions of the navigation task for future tasks. In this paper, we formalize the problem of minimizing the total expected cost to perform multiple start-to-goal navigation tasks on a roadmap by introducing the Learned Reactive Planning Problem. We propose a method that captures information from past executions to learn a motion policy to handle obstacles that the robot has seen before. We propose the LAMP framework, which integrates the generated motion policy with an existing navigation stack. Finally, an extensive set of experiments in simulated and real-world environments show that the proposed method outperforms the state-of-the-art algorithms by 10% to 40% in terms of expected time to travel from start to goal. We also evaluate the robustness of the proposed method in the presence of localization and mapping errors on a real robot.
△ Less
Submitted 3 December, 2020;
originally announced December 2020.
-
Interpretable Survival Prediction for Colorectal Cancer using Deep Learning
Authors:
Ellery Wulczyn,
David F. Steiner,
Melissa Moran,
Markus Plass,
Robert Reihs,
Fraser Tan,
Isabelle Flament-Auvigne,
Trissia Brown,
Peter Regitnig,
Po-Hsuan Cameron Chen,
Narayan Hegde,
Apaar Sadhwani,
Robert MacDonald,
Benny Ayalew,
Greg S. Corrado,
Lily H. Peng,
Daniel Tse,
Heimo Müller,
Zhaoyang Xu,
Yun Liu,
Martin C. Stumpe,
Kurt Zatloukal,
Craig H. Mermel
Abstract:
Deriving interpretable prognostic features from deep-learning-based prognostic histopathology models remains a challenge. In this study, we developed a deep learning system (DLS) for predicting disease specific survival for stage II and III colorectal cancer using 3,652 cases (27,300 slides). When evaluated on two validation datasets containing 1,239 cases (9,340 slides) and 738 cases (7,140 slide…
▽ More
Deriving interpretable prognostic features from deep-learning-based prognostic histopathology models remains a challenge. In this study, we developed a deep learning system (DLS) for predicting disease specific survival for stage II and III colorectal cancer using 3,652 cases (27,300 slides). When evaluated on two validation datasets containing 1,239 cases (9,340 slides) and 738 cases (7,140 slides) respectively, the DLS achieved a 5-year disease-specific survival AUC of 0.70 (95%CI 0.66-0.73) and 0.69 (95%CI 0.64-0.72), and added significant predictive value to a set of 9 clinicopathologic features. To interpret the DLS, we explored the ability of different human-interpretable features to explain the variance in DLS scores. We observed that clinicopathologic features such as T-category, N-category, and grade explained a small fraction of the variance in DLS scores (R2=18% in both validation sets). Next, we generated human-interpretable histologic features by clustering embeddings from a deep-learning based image-similarity model and showed that they explain the majority of the variance (R2 of 73% to 80%). Furthermore, the clustering-derived feature most strongly associated with high DLS scores was also highly prognostic in isolation. With a distinct visual appearance (poorly differentiated tumor cell clusters adjacent to adipose tissue), this feature was identified by annotators with 87.0-95.5% accuracy. Our approach can be used to explain predictions from a prognostic deep learning model and uncover potentially-novel prognostic features that can be reliably identified by people for future validation studies.
△ Less
Submitted 17 November, 2020;
originally announced November 2020.
-
Scaling advantage in quantum simulation of geometrically frustrated magnets
Authors:
Andrew D. King,
Jack Raymond,
Trevor Lanting,
Sergei V. Isakov,
Masoud Mohseni,
Gabriel Poulin-Lamarre,
Sara Ejtemaee,
William Bernoudy,
Isil Ozfidan,
Anatoly Yu. Smirnov,
Mauricio Reis,
Fabio Altomare,
Michael Babcock,
Catia Baron,
Andrew J. Berkley,
Kelly Boothby,
Paul I. Bunyk,
Holly Christiani,
Colin Enderud,
Bram Evert,
Richard Harris,
Emile Hoskinson,
Shuiyuan Huang,
Kais Jooya,
Ali Khodabandelou
, et al. (29 additional authors not shown)
Abstract:
The promise of quantum computing lies in harnessing programmable quantum devices for practical applications such as efficient simulation of quantum materials and condensed matter systems. One important task is the simulation of geometrically frustrated magnets in which topological phenomena can emerge from competition between quantum and thermal fluctuations. Here we report on experimental observa…
▽ More
The promise of quantum computing lies in harnessing programmable quantum devices for practical applications such as efficient simulation of quantum materials and condensed matter systems. One important task is the simulation of geometrically frustrated magnets in which topological phenomena can emerge from competition between quantum and thermal fluctuations. Here we report on experimental observations of relaxation in such simulations, measured on up to 1440 qubits with microsecond resolution. By initializing the system in a state with topological obstruction, we observe quantum annealing (QA) relaxation timescales in excess of one microsecond. Measurements indicate a dynamical advantage in the quantum simulation over the classical approach of path-integral Monte Carlo (PIMC) fixed-Hamiltonian relaxation with multiqubit cluster updates. The advantage increases with both system size and inverse temperature, exceeding a million-fold speedup over a CPU. This is an important piece of experimental evidence that in general, PIMC does not mimic QA dynamics for stoquastic Hamiltonians. The observed scaling advantage, for simulation of frustrated magnetism in quantum condensed matter, demonstrates that near-term quantum devices can be used to accelerate computational tasks of practical relevance.
△ Less
Submitted 8 November, 2019;
originally announced November 2019.
-
Fast Prototyping a Dialogue Comprehension System for Nurse-Patient Conversations on Symptom Monitoring
Authors:
Zhengyuan Liu,
Hazel Lim,
Nur Farah Ain Binte Suhaimi,
Shao Chuen Tong,
Sharon Ong,
Angela Ng,
Sheldon Lee,
Michael R. Macdonald,
Savitha Ramasamy,
Pavitra Krishnaswamy,
Wai Leng Chow,
Nancy F. Chen
Abstract:
Data for human-human spoken dialogues for research and development are currently very limited in quantity, variety, and sources; such data are even scarcer in healthcare. In this work, we investigate fast prototyping of a dialogue comprehension system by leveraging on minimal nurse-to-patient conversations. We propose a framework inspired by nurse-initiated clinical symptom monitoring conversation…
▽ More
Data for human-human spoken dialogues for research and development are currently very limited in quantity, variety, and sources; such data are even scarcer in healthcare. In this work, we investigate fast prototyping of a dialogue comprehension system by leveraging on minimal nurse-to-patient conversations. We propose a framework inspired by nurse-initiated clinical symptom monitoring conversations to construct a simulated human-human dialogue dataset, embodying linguistic characteristics of spoken interactions like thinking aloud, self-contradiction, and topic drift. We then adopt an established bidirectional attention pointer network on this simulated dataset, achieving more than 80% F1 score on a held-out test set from real-world nurse-to-patient conversations. The ability to automatically comprehend conversations in the healthcare domain by exploiting only limited data has implications for improving clinical workflows through red flag symptom detection and triaging capabilities. We demonstrate the feasibility for efficient and effective extraction, retrieval and comprehension of symptom checking information discussed in multi-turn human-human spoken conversations.
△ Less
Submitted 5 April, 2019; v1 submitted 8 March, 2019;
originally announced March 2019.
-
Microscope 2.0: An Augmented Reality Microscope with Real-time Artificial Intelligence Integration
Authors:
Po-Hsuan Cameron Chen,
Krishna Gadepalli,
Robert MacDonald,
Yun Liu,
Kunal Nagpal,
Timo Kohlberger,
Jeffrey Dean,
Greg S. Corrado,
Jason D. Hipp,
Martin C. Stumpe
Abstract:
The brightfield microscope is instrumental in the visual examination of both biological and physical samples at sub-millimeter scales. One key clinical application has been in cancer histopathology, where the microscopic assessment of the tissue samples is used for the diagnosis and staging of cancer and thus guides clinical therapy. However, the interpretation of these samples is inherently subje…
▽ More
The brightfield microscope is instrumental in the visual examination of both biological and physical samples at sub-millimeter scales. One key clinical application has been in cancer histopathology, where the microscopic assessment of the tissue samples is used for the diagnosis and staging of cancer and thus guides clinical therapy. However, the interpretation of these samples is inherently subjective, resulting in significant diagnostic variability. Moreover, in many regions of the world, access to pathologists is severely limited due to lack of trained personnel. In this regard, Artificial Intelligence (AI) based tools promise to improve the access and quality of healthcare. However, despite significant advances in AI research, integration of these tools into real-world cancer diagnosis workflows remains challenging because of the costs of image digitization and difficulties in deploying AI solutions. Here we propose a cost-effective solution to the integration of AI: the Augmented Reality Microscope (ARM). The ARM overlays AI-based information onto the current view of the sample through the optical pathway in real-time, enabling seamless integration of AI into the regular microscopy workflow. We demonstrate the utility of ARM in the detection of lymph node metastases in breast cancer and the identification of prostate cancer with a latency that supports real-time workflows. We anticipate that ARM will remove barriers towards the use of AI in microscopic analysis and thus improve the accuracy and efficiency of cancer diagnosis. This approach is applicable to other microscopy tasks and AI algorithms in the life sciences and beyond.
△ Less
Submitted 4 December, 2018; v1 submitted 21 November, 2018;
originally announced December 2018.
-
Development and Validation of a Deep Learning Algorithm for Improving Gleason Scoring of Prostate Cancer
Authors:
Kunal Nagpal,
Davis Foote,
Yun Liu,
Po-Hsuan,
Chen,
Ellery Wulczyn,
Fraser Tan,
Niels Olson,
Jenny L. Smith,
Arash Mohtashamian,
James H. Wren,
Greg S. Corrado,
Robert MacDonald,
Lily H. Peng,
Mahul B. Amin,
Andrew J. Evans,
Ankur R. Sangoi,
Craig H. Mermel,
Jason D. Hipp,
Martin C. Stumpe
Abstract:
For prostate cancer patients, the Gleason score is one of the most important prognostic factors, potentially determining treatment independent of the stage. However, Gleason scoring is based on subjective microscopic examination of tumor morphology and suffers from poor reproducibility. Here we present a deep learning system (DLS) for Gleason scoring whole-slide images of prostatectomies. Our syst…
▽ More
For prostate cancer patients, the Gleason score is one of the most important prognostic factors, potentially determining treatment independent of the stage. However, Gleason scoring is based on subjective microscopic examination of tumor morphology and suffers from poor reproducibility. Here we present a deep learning system (DLS) for Gleason scoring whole-slide images of prostatectomies. Our system was developed using 112 million pathologist-annotated image patches from 1,226 slides, and evaluated on an independent validation dataset of 331 slides, where the reference standard was established by genitourinary specialist pathologists. On the validation dataset, the mean accuracy among 29 general pathologists was 0.61. The DLS achieved a significantly higher diagnostic accuracy of 0.70 (p=0.002) and trended towards better patient risk stratification in correlations to clinical follow-up data. Our approach could improve the accuracy of Gleason scoring and subsequent therapy decisions, particularly where specialist expertise is unavailable. The DLS also goes beyond the current Gleason system to more finely characterize and quantitate tumor morphology, providing opportunities for refinement of the Gleason system itself.
△ Less
Submitted 15 November, 2018;
originally announced November 2018.
-
A Multimodal Corpus of Expert Gaze and Behavior during Phonetic Segmentation Tasks
Authors:
Arif Khan,
Ingmar Steiner,
Yusuke Sugano,
Andreas Bulling,
Ross Macdonald
Abstract:
Phonetic segmentation is the process of splitting speech into distinct phonetic units. Human experts routinely perform this task manually by analyzing auditory and visual cues using analysis software, which is an extremely time-consuming process. Methods exist for automatic segmentation, but these are not always accurate enough. In order to improve automatic segmentation, we need to model it as cl…
▽ More
Phonetic segmentation is the process of splitting speech into distinct phonetic units. Human experts routinely perform this task manually by analyzing auditory and visual cues using analysis software, which is an extremely time-consuming process. Methods exist for automatic segmentation, but these are not always accurate enough. In order to improve automatic segmentation, we need to model it as close to the manual segmentation as possible. This corpus is an effort to capture the human segmentation behavior by recording experts performing a segmentation task. We believe that this data will enable us to highlight the important aspects of manual segmentation, which can be used in automatic segmentation to improve its accuracy.
△ Less
Submitted 11 May, 2018; v1 submitted 13 December, 2017;
originally announced December 2017.
-
Covert Communication over Classical-Quantum Channels
Authors:
Michael S. Bullock,
Azadeh Sheikholeslami,
Mehrdad Tahmasbi,
Robert C. Macdonald,
Saikat Guha,
Boulat A. Bash
Abstract:
We investigate covert communication over general memoryless classical-quantum channels with fixed finite-size input alphabets. We show that the square root law (SRL) governs covert communication in this setting when product of $n$ input states is used: $L_{\rm SRL}\sqrt{n}+o(\sqrt{n})$ covert bits (but no more) can be reliably transmitted in $n$ uses of classical-quantum channel, where…
▽ More
We investigate covert communication over general memoryless classical-quantum channels with fixed finite-size input alphabets. We show that the square root law (SRL) governs covert communication in this setting when product of $n$ input states is used: $L_{\rm SRL}\sqrt{n}+o(\sqrt{n})$ covert bits (but no more) can be reliably transmitted in $n$ uses of classical-quantum channel, where $L_{\rm SRL}>0$ is a channel-dependent constant that we call covert capacity. We also show that ensuring covertness requires $J_{\rm SRL}\sqrt{n}+o(\sqrt{n})$ bits secret shared by the communicating parties prior to transmission, where $J_{\rm SRL}\geq0$ is a channel-dependent constant. We assume a quantum-powerful adversary that can perform an arbitrary joint (entangling) measurement on all $n$ channel uses. We determine the single-letter expressions for $L_{\rm SRL}$ and $J_{\rm SRL}$, and establish conditions when $J_{\rm SRL}=0$ (i.e., no pre-shared secret is needed). Finally, we evaluate the scenarios where covert communication is not governed by the SRL.
△ Less
Submitted 21 July, 2023; v1 submitted 25 January, 2016;
originally announced January 2016.
