-
DaedalusData: Exploration, Knowledge Externalization and Labeling of Particles in Medical Manufacturing -- A Design Study
Authors:
Alexander Wyss,
Gabriela Morgenshtern,
Amanda Hirsch-Hüsler,
Jürgen Bernard
Abstract:
In medical diagnostics of both early disease detection and routine patient care, particle-based contamination of in-vitro diagnostics consumables poses a significant threat to patients. Objective data-driven decision-making on the severity of contamination is key for reducing patient risk, while saving time and cost in quality assessment. Our collaborators introduced us to their quality control pr…
▽ More
In medical diagnostics of both early disease detection and routine patient care, particle-based contamination of in-vitro diagnostics consumables poses a significant threat to patients. Objective data-driven decision-making on the severity of contamination is key for reducing patient risk, while saving time and cost in quality assessment. Our collaborators introduced us to their quality control process, including particle data acquisition through image recognition, feature extraction, and attributes reflecting the production context of particles. Shortcomings in the current process are limitations in exploring thousands of images, data-driven decision making, and ineffective knowledge externalization. Following the design study methodology, our contributions are a characterization of the problem space and requirements, the development and validation of DaedalusData, a comprehensive discussion of our study's learnings, and a generalizable framework for knowledge externalization. DaedalusData is a visual analytics system that enables domain experts to explore particle contamination patterns, label particles in label alphabets, and externalize knowledge through semi-supervised label-informed data projections. The results of our case study and user study show high usability of DaedalusData and its efficient support of experts in generating comprehensive overviews of thousands of particles, labeling of large quantities of particles, and externalizing knowledge to augment the dataset further. Reflecting on our approach, we discuss insights on dataset augmentation via human knowledge externalization, and on the scalability and trade-offs that come with the adoption of this approach in practice.
△ Less
Submitted 8 August, 2024;
originally announced August 2024.
-
A Data-Centric Methodology and Task Typology for Time-Stamped Event Sequences
Authors:
Yasara Peiris,
Clara-Maria Barth,
Elaine M. Huang,
Jürgen Bernard
Abstract:
Task abstractions and taxonomic structures for tasks are useful for designers of interactive data analysis approaches, serving as design targets and evaluation criteria alike. For individual data types, dataset-specific taxonomic structures capture unique data characteristics, while being generalizable across application domains. The creation of dataset-centric but domain-agnostic taxonomic struct…
▽ More
Task abstractions and taxonomic structures for tasks are useful for designers of interactive data analysis approaches, serving as design targets and evaluation criteria alike. For individual data types, dataset-specific taxonomic structures capture unique data characteristics, while being generalizable across application domains. The creation of dataset-centric but domain-agnostic taxonomic structures is difficult, especially if best practices for a focused data type are still missing, observing experts is not feasible, and means for reflection and generalization are scarce. We discovered this need for methodological support when working with time-stamped event sequences, a datatype that has not yet been fully systematically studied in visualization research. To address this shortcoming, we present a methodology that enables researchers to abstract tasks and build dataset-centric taxonomic structures in five phases (data collection, coding, task categorization, task synthesis, and action-target(criterion) crosscut). We validate the methodology by applying it to time-stamped event sequences and present a task typology that uses triples as a novel language of description for tasks: (1) action, (2) data target, and (3) data criterion. We further evaluate the descriptive power of the typology with a real-world case on cybersecurity.
△ Less
Submitted 21 September, 2022;
originally announced September 2022.
-
A Design Space for Explainable Ranking and Ranking Models
Authors:
I. Al Hazwani,
J. Schmid,
M. Sachdeva,
J. Bernard
Abstract:
Item ranking systems support users in multi-criteria decision-making tasks. Users need to trust rankings and ranking algorithms to reflect user preferences nicely while avoiding systematic errors and biases. However, today only few approaches help end users, model developers, and analysts to explain rankings. We report on the study of explanation approaches from the perspectives of recommender sys…
▽ More
Item ranking systems support users in multi-criteria decision-making tasks. Users need to trust rankings and ranking algorithms to reflect user preferences nicely while avoiding systematic errors and biases. However, today only few approaches help end users, model developers, and analysts to explain rankings. We report on the study of explanation approaches from the perspectives of recommender systems, explainable AI, and visualization research and propose the first cross-domain design space for explainers of item rankings. In addition, we leverage the descriptive power of the design space to characterize a) existing explainers and b) three main user groups involved in ranking explanation tasks. The generative power of the design space is a means for future designers and developers to create more target-oriented solutions in this only weakly exploited space.
△ Less
Submitted 27 May, 2022;
originally announced May 2022.
-
Node package manager's dependency network robustness
Authors:
Andrej Hafner,
Anže Mur,
Jaka Bernard
Abstract:
The robustness of npm dependency network is a crucial property, since many projects and web applications heavily rely on the functionalities of packages, especially popular ones that have many dependant packages. In the past, there have been instances where the removal or update of certain npm packages has caused widespread chaos and web-page downtime on the internet. Our goal is to track the netw…
▽ More
The robustness of npm dependency network is a crucial property, since many projects and web applications heavily rely on the functionalities of packages, especially popular ones that have many dependant packages. In the past, there have been instances where the removal or update of certain npm packages has caused widespread chaos and web-page downtime on the internet. Our goal is to track the network's resilience to such occurrences through time and figure out whether the state of the network is trending towards a more robust structure. We show that the network is not robust to targeted attacks, since a security risk in a few crucial nodes affects a large part of the network. Because such packages are often backed up by serious communities with high standards, the issue is not alarming and is a consequence of power law distribution of the network. The current trend in average number of dependencies and effect of important nodes on the rest of the network is decreasing, which further improves the resilience and sets a positive path in development. Furthermore, we show that communities form around the most important packages, although they do not conform well to the common community definition using modularity. We also provide guidelines for package development that increases the robustness of the network and reduces the possibility of introducing security risks.
△ Less
Submitted 22 October, 2021;
originally announced October 2021.
-
Does the Layout Really Matter? A Study on Visual Model Accuracy Estimation
Authors:
Nicolas Grossmann,
Jürgen Bernard,
Michael Sedlmair,
Manuela Waldner
Abstract:
In visual interactive labeling, users iteratively assign labels to data items until the machine model reaches an acceptable accuracy. A crucial step of this process is to inspect the model's accuracy and decide whether it is necessary to label additional elements. In scenarios with no or very little labeled data, visual inspection of the predictions is required. Similarity-preserving scatterplots…
▽ More
In visual interactive labeling, users iteratively assign labels to data items until the machine model reaches an acceptable accuracy. A crucial step of this process is to inspect the model's accuracy and decide whether it is necessary to label additional elements. In scenarios with no or very little labeled data, visual inspection of the predictions is required. Similarity-preserving scatterplots created through a dimensionality reduction algorithm are a common visualization that is used in these cases. Previous studies investigated the effects of layout and image complexity on tasks like labeling. However, model evaluation has not been studied systematically. We present the results of an experiment studying the influence of image complexity and visual grouping of images on model accuracy estimation. We found that users outperform traditional automated approaches when estimating a model's accuracy. Furthermore, while the complexity of images impacts the overall performance, the layout of the items in the plot has little to no effect on estimations.
△ Less
Submitted 14 October, 2021;
originally announced October 2021.
-
Visualizing Graph Neural Networks with CorGIE: Corresponding a Graph to Its Embedding
Authors:
Zipeng Liu,
Yang Wang,
Jürgen Bernard,
Tamara Munzner
Abstract:
Graph neural networks (GNNs) are a class of powerful machine learning tools that model node relations for making predictions of nodes or links. GNN developers rely on quantitative metrics of the predictions to evaluate a GNN, but similar to many other neural networks, it is difficult for them to understand if the GNN truly learns characteristics of a graph as expected. We propose an approach to co…
▽ More
Graph neural networks (GNNs) are a class of powerful machine learning tools that model node relations for making predictions of nodes or links. GNN developers rely on quantitative metrics of the predictions to evaluate a GNN, but similar to many other neural networks, it is difficult for them to understand if the GNN truly learns characteristics of a graph as expected. We propose an approach to corresponding an input graph to its node embedding (aka latent space), a common component of GNNs that is later used for prediction. We abstract the data and tasks, and develop an interactive multi-view interface called CorGIE to instantiate the abstraction. As the key function in CorGIE, we propose the K-hop graph layout to show topological neighbors in hops and their clustering structure. To evaluate the functionality and usability of CorGIE, we present how to use CorGIE in two usage scenarios, and conduct a case study with five GNN experts.
△ Less
Submitted 8 November, 2021; v1 submitted 24 June, 2021;
originally announced June 2021.
-
Automated Grading of Anatomical Objective Structured Practical Exams Using Decision Trees
Authors:
Jason Bernard,
Ranil Sonnadara,
Anthony N. Saraco,
Josh P. Mitchell,
Alex B. Bak,
Ilana Bayer,
Bruce C. Wainman
Abstract:
An Objective Structured Practical Examination (OSPE) is an effective and robust, but resource-intensive, means of evaluating anatomical knowledge. Since most OSPEs employ short answer or fill-in-the-blank style questions, the format requires many people familiar with the content to mark the exams. However, the increasing prevalence of online delivery for anatomy and physiology courses could result…
▽ More
An Objective Structured Practical Examination (OSPE) is an effective and robust, but resource-intensive, means of evaluating anatomical knowledge. Since most OSPEs employ short answer or fill-in-the-blank style questions, the format requires many people familiar with the content to mark the exams. However, the increasing prevalence of online delivery for anatomy and physiology courses could result in students losing the OSPE practice that they would receive in face-to-face learning sessions. The purpose of this study was to test the accuracy of Decision Trees (DTs) in marking OSPE questions as a potential first step to creating an intelligent, online OSPE tutoring system. The study used the results of the winter 2020 semester final OSPE from McMaster University's anatomy and physiology course in the Faculty of Health Sciences (HTHSCI 2FF3/2LL3/1D06) as the data set. Ninety percent of the data set was used in a 10-fold validation algorithm to train a DT for each of the 54 questions. Each DT was comprised of unique words that appeared in correct, student-written answers. The remaining 10% of the data set was marked by the generated DTs. When the answers marked by the DT were compared to the answers marked by staff and faculty, the DT achieved an average accuracy of 94.49% across all 54 questions. This suggests that machine learning algorithms such as DTs are a highly effective option for OSPE grading and are suitable for the development of an intelligent, online OSPE tutoring system.
△ Less
Submitted 1 June, 2021;
originally announced June 2021.
-
Stochastic L-system Inference from Multiple String Sequence Inputs
Authors:
Jason Bernard,
Ian McQuillan
Abstract:
Lindenmayer systems (L-systems) are a grammar system that consist of string rewriting rules. The rules replace every symbol in a string in parallel with a successor to produce the next string, and this procedure iterates. In a stochastic context-free L-system (S0L-system), every symbol may have one or more rewriting rule, each with an associated probability of selection. Properly constructed rewri…
▽ More
Lindenmayer systems (L-systems) are a grammar system that consist of string rewriting rules. The rules replace every symbol in a string in parallel with a successor to produce the next string, and this procedure iterates. In a stochastic context-free L-system (S0L-system), every symbol may have one or more rewriting rule, each with an associated probability of selection. Properly constructed rewriting rules have been found to be useful for modeling and simulating some natural and human engineered processes where each derived string describes a step in the simulation. Typically, processes are modeled by experts who meticulously construct the rules based on measurements or domain knowledge of the process. This paper presents an automated approach to finding stochastic L-systems, given a set of string sequences as input. The implemented tool is called the Plant Model Inference Tool for S0L-systems (PMIT-S0L). PMIT-S0L is evaluated using 960 procedurally generated S0L-systems in a test suite, which are each used to generate input strings, and PMIT-S0L is then used to infer the system from only the sequences. The evaluation shows that PMIT-S0L infers S0L-systems with up to 9 rewriting rules each in under 12 hours. Additionally, it is found that 3 sequences of strings is sufficient to find the correct original rewriting rules in 100% of the cases in the test suite, and 6 sequences of strings reduces the difference in the associated probabilities to approximately 1% or less.
△ Less
Submitted 29 January, 2020;
originally announced January 2020.
-
ConfusionFlow: A model-agnostic visualization for temporal analysis of classifier confusion
Authors:
Andreas Hinterreiter,
Peter Ruch,
Holger Stitz,
Martin Ennemoser,
Jürgen Bernard,
Hendrik Strobelt,
Marc Streit
Abstract:
Classifiers are among the most widely used supervised machine learning algorithms. Many classification models exist, and choosing the right one for a given task is difficult. During model selection and debugging, data scientists need to assess classifiers' performances, evaluate their learning behavior over time, and compare different models. Typically, this analysis is based on single-number perf…
▽ More
Classifiers are among the most widely used supervised machine learning algorithms. Many classification models exist, and choosing the right one for a given task is difficult. During model selection and debugging, data scientists need to assess classifiers' performances, evaluate their learning behavior over time, and compare different models. Typically, this analysis is based on single-number performance measures such as accuracy. A more detailed evaluation of classifiers is possible by inspecting class errors. The confusion matrix is an established way for visualizing these class errors, but it was not designed with temporal or comparative analysis in mind. More generally, established performance analysis systems do not allow a combined temporal and comparative analysis of class-level information. To address this issue, we propose ConfusionFlow, an interactive, comparative visualization tool that combines the benefits of class confusion matrices with the visualization of performance characteristics over time. ConfusionFlow is model-agnostic and can be used to compare performances for different model types, model architectures, and/or training and test datasets. We demonstrate the usefulness of ConfusionFlow in a case study on instance selection strategies in active learning. We further assess the scalability of ConfusionFlow and present a use case in the context of neural network pruning.
△ Less
Submitted 2 July, 2020; v1 submitted 2 October, 2019;
originally announced October 2019.
-
Speculative Execution for Guided Visual Analytics
Authors:
Fabian Sperrle,
Jürgen Bernard,
Michael Sedlmair,
Daniel Keim,
Mennatallah El-Assady
Abstract:
We propose the concept of Speculative Execution for Visual Analytics and discuss its effectiveness for model exploration and optimization. Speculative Execution enables the automatic generation of alternative, competing model configurations that do not alter the current model state unless explicitly confirmed by the user. These alternatives are computed based on either user interactions or model q…
▽ More
We propose the concept of Speculative Execution for Visual Analytics and discuss its effectiveness for model exploration and optimization. Speculative Execution enables the automatic generation of alternative, competing model configurations that do not alter the current model state unless explicitly confirmed by the user. These alternatives are computed based on either user interactions or model quality measures and can be explored using delta-visualizations. By automatically proposing modeling alternatives, systems employing Speculative Execution can shorten the gap between users and models, reduce the confirmation bias and speed up optimization processes. In this paper, we have assembled five application scenarios showcasing the potential of Speculative Execution, as well as a potential for further research.
△ Less
Submitted 7 August, 2019;
originally announced August 2019.
-
Techniques for Inferring Context-Free Lindenmayer Systems With Genetic Algorithm
Authors:
Jason Bernard,
Ian McQuillan
Abstract:
Lindenmayer systems (L-systems) are a formal grammar system, where the most notable feature is a set of rewriting rules that are used to replace every symbol in a string in parallel; by repeating this process, a sequence of strings is produced. Some symbols in the strings may be interpreted as instructions for simulation software. Thus, the sequence can be used to model the steps of a process. Cur…
▽ More
Lindenmayer systems (L-systems) are a formal grammar system, where the most notable feature is a set of rewriting rules that are used to replace every symbol in a string in parallel; by repeating this process, a sequence of strings is produced. Some symbols in the strings may be interpreted as instructions for simulation software. Thus, the sequence can be used to model the steps of a process. Currently, creating an L-system for a specific process is done by hand by experts through much effort. The inductive inference problem attempts to infer an L-system from such a sequence of strings generated by an unknown system; this can be thought of as an intermediate step to inferring from a sequence of images. This paper evaluates and analyzes different genetic algorithm encoding schemes and mathematical properties for the L-system inductive inference problem. A new tool, the Plant Model Inference Tool for Context-Free L-systems (PMIT-D0L) is implemented based on these techniques. PMIT-D0L has been successfully evaluated on 28 known L-systems, with alphabets up to 31 symbols and a total sum of 281 symbols across the rewriting rules. PMIT-D0L can infer even the largest of these L-systems in less than a few seconds.
△ Less
Submitted 29 April, 2021; v1 submitted 15 May, 2019;
originally announced June 2019.
-
An Approximation-based Approach for the Random Exploration of Large Models
Authors:
Julien Bernard,
Pierre-Cyrille Héam,
Olga Kouchnarenko
Abstract:
System modeling is a classical approach to ensure their reliability since it is suitable both for a formal verification and for software testing techniques. In the context of model-based testing an approach combining random testing and coverage based testing has been recently introduced [9]. However, this approach is not tractable on quite large models. In this paper we show how to use statistical…
▽ More
System modeling is a classical approach to ensure their reliability since it is suitable both for a formal verification and for software testing techniques. In the context of model-based testing an approach combining random testing and coverage based testing has been recently introduced [9]. However, this approach is not tractable on quite large models. In this paper we show how to use statistical approximations to make the approach work on larger models. Experimental results, on models of communicating protocols, are provided; they are very promising, both for the computation time and for the quality of the generated test suites.
△ Less
Submitted 13 June, 2018;
originally announced June 2018.
-
New Techniques for Inferring L-Systems Using Genetic Algorithm
Authors:
Jason Bernard,
Ian McQuillan
Abstract:
Lindenmayer systems (L-systems) are a formal grammar system that iteratively rewrites all symbols of a string, in parallel. When visualized with a graphical interpretation, the images have self-similar shapes that appear frequently in nature, and they have been particularly successful as a concise, reusable technique for simulating plants. The L-system inference problem is to find an L-system to s…
▽ More
Lindenmayer systems (L-systems) are a formal grammar system that iteratively rewrites all symbols of a string, in parallel. When visualized with a graphical interpretation, the images have self-similar shapes that appear frequently in nature, and they have been particularly successful as a concise, reusable technique for simulating plants. The L-system inference problem is to find an L-system to simulate a given plant. This is currently done mainly by experts, but this process is limited by the availability of experts, the complexity that may be solved by humans, and time. This paper introduces the Plant Model Inference Tool (PMIT) that infers deterministic context-free L-systems from an initial sequence of strings generated by the system using a genetic algorithm. PMIT is able to infer more complex systems than existing approaches. Indeed, while existing approaches are limited to L-systems with a total sum of 20 combined symbols in the productions, PMIT can infer almost all L-systems tested where the total sum is 140 symbols. This was validated using a test bed of 28 previously developed L-system models, in addition to models created artificially by bootstrapping larger models.
△ Less
Submitted 4 December, 2017; v1 submitted 30 November, 2017;
originally announced December 2017.
-
Visual-Interactive Similarity Search for Complex Objects by Example of Soccer Player Analysis
Authors:
Jürgen Bernard,
Christian Ritter,
David Sessler,
Matthias Zeppelzauer,
Jörn Kohlhammer,
Dieter Fellner
Abstract:
The definition of similarity is a key prerequisite when analyzing complex data types in data mining, information retrieval, or machine learning. However, the meaningful definition is often hampered by the complexity of data objects and particularly by different notions of subjective similarity latent in targeted user groups. Taking the example of soccer players, we present a visual-interactive sys…
▽ More
The definition of similarity is a key prerequisite when analyzing complex data types in data mining, information retrieval, or machine learning. However, the meaningful definition is often hampered by the complexity of data objects and particularly by different notions of subjective similarity latent in targeted user groups. Taking the example of soccer players, we present a visual-interactive system that learns users' mental models of similarity. In a visual-interactive interface, users are able to label pairs of soccer players with respect to their subjective notion of similarity. Our proposed similarity model automatically learns the respective concept of similarity using an active learning strategy. A visual-interactive retrieval technique is provided to validate the model and to execute downstream retrieval tasks for soccer player analysis. The applicability of the approach is demonstrated in different evaluation strategies, including usage scenarions and cross-validation tests.
△ Less
Submitted 9 March, 2017;
originally announced March 2017.
