-
ClimDetect: A Benchmark Dataset for Climate Change Detection and Attribution
Authors:
Sungduk Yu,
Brian L. White,
Anahita Bhiwandiwalla,
Musashi Hinck,
Matthew Lyle Olson,
Tung Nguyen,
Vasudev Lal
Abstract:
Detecting and attributing temperature increases due to climate change is crucial for understanding global warming and guiding adaptation strategies. The complexity of distinguishing human-induced climate signals from natural variability has challenged traditional detection and attribution (D&A) approaches, which seek to identify specific "fingerprints" in climate response variables. Deep learning…
▽ More
Detecting and attributing temperature increases due to climate change is crucial for understanding global warming and guiding adaptation strategies. The complexity of distinguishing human-induced climate signals from natural variability has challenged traditional detection and attribution (D&A) approaches, which seek to identify specific "fingerprints" in climate response variables. Deep learning offers potential for discerning these complex patterns in expansive spatial datasets. However, lack of standard protocols has hindered consistent comparisons across studies. We introduce ClimDetect, a standardized dataset of over 816k daily climate snapshots, designed to enhance model accuracy in identifying climate change signals. ClimDetect integrates various input and target variables used in past research, ensuring comparability and consistency. We also explore the application of vision transformers (ViT) to climate data, a novel and modernizing approach in this context. Our open-access data and code serve as a benchmark for advancing climate science through improved model evaluations. ClimDetect is publicly accessible via Huggingface dataet respository at: https://huggingface.co/datasets/ClimDetect/ClimDetect.
△ Less
Submitted 28 August, 2024;
originally announced August 2024.
-
Satisfaction and Regret in Stackelberg Games
Authors:
Langford White,
Duong Nguyen,
Hung Nguyen
Abstract:
This paper introduces the new concept of (follower) satisfaction in Stackelberg games and compares the standard Stackelberg game with its satisfaction version. Simulation results are presented which suggest that the follower adopting satisfaction generally increases leader utility. This important new result is proven for the case where leader strategies to commit to are restricted to be determinis…
▽ More
This paper introduces the new concept of (follower) satisfaction in Stackelberg games and compares the standard Stackelberg game with its satisfaction version. Simulation results are presented which suggest that the follower adopting satisfaction generally increases leader utility. This important new result is proven for the case where leader strategies to commit to are restricted to be deterministic (pure strategies). The paper then addresses the application of regret based algorithms to the Stackelberg problem. Although it is known that the follower adopts a no-regret position in a Stackelberg solution, this is not generally the case for the leader. The report examines the convergence behaviour of unconditional and conditional regret matching (RM) algorithms in the Stackelberg setting. The paper shows that, in the examples considered, that these algorithms either converge to any pure Nash equilibria for the simultaneous move game, or to some mixed strategies which do not have the "no-regret" property. In one case, convergence of the conditional RM algorithm over both players to a solution "close" to the Stackelberg case was observed. The paper argues that further research in this area, in particular when applied in the satisfaction setting could be fruitful.
△ Less
Submitted 21 August, 2024;
originally announced August 2024.
-
Modal Effect Types
Authors:
Wenhao Tang,
Leo White,
Stephen Dolan,
Daniel Hillerström,
Sam Lindley,
Anton Lorenzen
Abstract:
We propose a novel type system for effects and handlers using modal types. Conventional effect systems attach effects to function types, which can lead to verbose effect-polymorphic types, especially for higher-order functions. Our modal effect system provides succinct types for higher-order first-class functions without losing modularity and reusability. The core idea is to decouple effects from…
▽ More
We propose a novel type system for effects and handlers using modal types. Conventional effect systems attach effects to function types, which can lead to verbose effect-polymorphic types, especially for higher-order functions. Our modal effect system provides succinct types for higher-order first-class functions without losing modularity and reusability. The core idea is to decouple effects from function types and instead to track effects through relative and absolute modalities, which represent transformations on the ambient effects provided by the context.
We formalise the idea of modal effect types in a multimodal System F-style core calculus Met with effects and handlers. Met supports modular effectful programming via modalities without relying on effect variables. We encode a practical fragment of a conventional row-based effect system with effect polymorphism, which captures most common use-cases, into Met in order to formally demonstrate the expressive power of modal effect types. To recover the full power of conventional effect systems beyond this fragment, we seamlessly extend Met to Mete with effect variables. We propose a surface language Metel for Mete with a sound and complete type inference algorithm inspired by FreezeML.
△ Less
Submitted 16 July, 2024;
originally announced July 2024.
-
A pilot protocol and cohort for the investigation of non-pathological variability in speech
Authors:
Nicholas Cummins,
Lauren L. White,
Zahia Rahman,
Catriona Lucas,
Tian Pan,
Ewan Carr,
Faith Matcham,
Johnny Downs,
Richard J. Dobson,
Judith Dineley
Abstract:
Background Speech-based biomarkers have potential as a means for regular, objective assessment of symptom severity, remotely and in-clinic in combination with advanced analytical models. However, the complex nature of speech and the often subtle changes associated with health mean that findings are highly dependent on methodological and cohort choices. These are often not reported adequately in st…
▽ More
Background Speech-based biomarkers have potential as a means for regular, objective assessment of symptom severity, remotely and in-clinic in combination with advanced analytical models. However, the complex nature of speech and the often subtle changes associated with health mean that findings are highly dependent on methodological and cohort choices. These are often not reported adequately in studies investigating speech-based health assessment Objective To develop and apply an exemplar protocol to generate a pilot dataset of healthy speech with detailed metadata for the assessment of factors in the speech recording-analysis pipeline, including device choice, speech elicitation task and non-pathological variability. Methods We developed our collection protocol and choice of exemplar speech features based on a thematic literature review. Our protocol includes the elicitation of three different speech types. With a focus towards remote applications, we also choose to collect speech with three different microphone types. We developed a pipeline to extract a set of 14 exemplar speech features. Results We collected speech from 28 individuals three times in one day, repeated at the same times 8-11 weeks later, and from 25 healthy individuals three times in one week. Participant characteristics collected included sex, age, native language status and voice use habits of the participant. A preliminary set of 14 speech features covering timing, prosody, voice quality, articulation and spectral moment characteristics were extracted that provide a resource of normative values. Conclusions There are multiple methodological factors involved in the collection, processing and analysis of speech recordings. Consistent reporting and greater harmonisation of study protocols are urgently required to aid the translation of speech processing into clinical research and practice.
△ Less
Submitted 11 June, 2024;
originally announced June 2024.
-
Embedding Differential Dynamic Logic in PVS
Authors:
J. Tanner Slagel,
Mariano Moscato,
Lauren White,
César A. Muñoz,
Swee Balachandran,
Aaron Dutle
Abstract:
Differential dynamic logic (dL) is a formal framework for specifying and reasoning about hybrid systems, i.e., dynamical systems that exhibit both continuous and discrete behaviors. These kinds of systems arise in many safety- and mission-critical applications. This paper presents a formalization of dL in the Prototype Verification System (PVS) that includes the semantics of hybrid programs and d…
▽ More
Differential dynamic logic (dL) is a formal framework for specifying and reasoning about hybrid systems, i.e., dynamical systems that exhibit both continuous and discrete behaviors. These kinds of systems arise in many safety- and mission-critical applications. This paper presents a formalization of dL in the Prototype Verification System (PVS) that includes the semantics of hybrid programs and dL's proof calculus. The formalization embeds dL into the PVS logic, resulting in a version of dL whose proof calculus is not only formally verified, but is also available for the verification of hybrid programs within PVS itself. This embedding, called Plaidypvs (Properly Assured Implementation of dL for Hybrid Program Verification and Specification), supports standard dL style proofs, but further leverages the capabilities of PVS to allow reasoning about entire classes of hybrid programs. The embedding also allows the user to import the well-established definitions and mathematical theories available in PVS.
△ Less
Submitted 23 April, 2024;
originally announced April 2024.
-
Automatic Defect Detection in Sewer Network Using Deep Learning Based Object Detector
Authors:
Bach Ha,
Birgit Schalter,
Laura White,
Joachim Koehler
Abstract:
Maintaining sewer systems in large cities is important, but also time and effort consuming, because visual inspections are currently done manually. To reduce the amount of aforementioned manual work, defects within sewer pipes should be located and classified automatically. In the past, multiple works have attempted solving this problem using classical image processing, machine learning, or a comb…
▽ More
Maintaining sewer systems in large cities is important, but also time and effort consuming, because visual inspections are currently done manually. To reduce the amount of aforementioned manual work, defects within sewer pipes should be located and classified automatically. In the past, multiple works have attempted solving this problem using classical image processing, machine learning, or a combination of those. However, each provided solution only focus on detecting a limited set of defect/structure types, such as fissure, root, and/or connection. Furthermore, due to the use of hand-crafted features and small training datasets, generalization is also problematic. In order to overcome these deficits, a sizable dataset with 14.7 km of various sewer pipes were annotated by sewer maintenance experts in the scope of this work. On top of that, an object detector (EfficientDet-D0) was trained for automatic defect detection. From the result of several expermients, peculiar natures of defects in the context of object detection, which greatly effect annotation and training process, are found and discussed. At the end, the final detector was able to detect 83% of defects in the test set; out of the missing 17%, only 0.77% are very severe defects. This work provides an example of applying deep learning-based object detection into an important but quiet engineering field. It also gives some practical pointers on how to annotate peculiar "object", such as defects.
△ Less
Submitted 9 April, 2024;
originally announced April 2024.
-
A Preliminary Exploration of the Disruption of a Generative AI Systems: Faculty/Staff and Student Perceptions of ChatGPT and its Capability of Completing Undergraduate Engineering Coursework
Authors:
Lance White,
Trini Balart,
Sara Amani,
Kristi J. Shryock,
Karan L. Watson
Abstract:
The authors of this study aim to assess the capabilities of the OpenAI ChatGPT tool to understand just how effective such a system might be for students to utilize in their studies as well as deepen understanding of faculty/staff and student perceptions about ChatGPT in general. The purpose of what is learned from the study is to continue the design of a model to facilitate the development of facu…
▽ More
The authors of this study aim to assess the capabilities of the OpenAI ChatGPT tool to understand just how effective such a system might be for students to utilize in their studies as well as deepen understanding of faculty/staff and student perceptions about ChatGPT in general. The purpose of what is learned from the study is to continue the design of a model to facilitate the development of faculty for becoming adept at embracing change, the DANCE model (Designing Adaptations for the Next Changes in Education). This model is used in this study to help faculty with examining the impact that a disruptive new tool, such as ChatGPT, can pose for the learning environment.
The authors analyzed the performance of ChatGPT used to complete course assignments at a variety of levels by novice engineering students working as research assistants. Those completed works have been assessed by the faculty who created those assignments to understand how these completed assignments might compare with the performance of a typical student. A set of surveys conducted by the authors of this work are discussed where students, faculty, and staff respondents in March of 2023 addressed their perceptions of ChatGPT (A follow-up survey is being administered now, February 2024). These survey instruments were analyzed, and the data visualized in this work to bring attention to relevant findings by the researchers. This work reports the findings of the researchers with the purpose of sharing the current state of this work at Texas A&M University with the intention to provide insights to scholars both at our own institution and around the world. This work is not intended to be a finished work but reports these findings with full transparency that this work is currently continuing as the researchers gather new data and develop and validate various measurement instruments.
△ Less
Submitted 3 March, 2024;
originally announced March 2024.
-
Bayesian Rationality in Satisfaction Games
Authors:
Langford White,
Oskar Rynkiewicz,
Duong Nguyen,
Hung Nguyen
Abstract:
We introduce a new paradigm for game theory -- Bayesian satisfaction. This novel approach is a synthesis of the idea of Bayesian rationality introduced by Aumann, and satisfaction games. The concept of Bayesian rationality for which, in part, Robert Aumann was awarded the Nobel Prize in 2005, is concerned with players in a game acting in their own best interest given a subjective knowledge of the…
▽ More
We introduce a new paradigm for game theory -- Bayesian satisfaction. This novel approach is a synthesis of the idea of Bayesian rationality introduced by Aumann, and satisfaction games. The concept of Bayesian rationality for which, in part, Robert Aumann was awarded the Nobel Prize in 2005, is concerned with players in a game acting in their own best interest given a subjective knowledge of the other players' behaviours as represented by a probability distribution. Satisfaction games have emerged in the engineering literature as a way of modelling competitive interactions in resource allocation problems where players seek to attain a specified level of utility, rather than trying to maximise utility. In this paper, we explore the relationship between optimality in Aumann's sense (correlated equilibria), and satisfaction in games. We show that correlated equilibria in a satisfaction game represent stable outcomes in which no player can increase their probability of satisfaction by unilateral deviation from the specified behaviour. Thus, we propose a whole new class of equilibrium outcomes in satisfaction games which include existing notions of equilibria in such games. Iterative algorithms for computing such equilibria based on the existing ideas of regret matching are presented and interpreted within the satisfaction framework. Numerical examples of resource allocation are presented to illustrate the behaviour of these algorithms. A notable feature of these algorithms is that they almost always find equilibrium outcomes whereas existing approaches in satisfaction games may not.
△ Less
Submitted 25 July, 2023;
originally announced July 2023.
-
Social Optimum Equilibrium Selection for Distributed Multi-Agent Optimization
Authors:
Duong Nguyen,
Langford White,
Hung Nguyen
Abstract:
We study the open question of how players learn to play a social optimum pure-strategy Nash equilibrium (PSNE) through repeated interactions in general-sum coordination games. A social optimum of a game is the stable Pareto-optimal state that provides a maximum return in the sum of all players' payoffs (social welfare) and always exists. We consider finite repeated games where each player only has…
▽ More
We study the open question of how players learn to play a social optimum pure-strategy Nash equilibrium (PSNE) through repeated interactions in general-sum coordination games. A social optimum of a game is the stable Pareto-optimal state that provides a maximum return in the sum of all players' payoffs (social welfare) and always exists. We consider finite repeated games where each player only has access to its own utility (or payoff) function but is able to exchange information with other players. We develop a novel regret matching (RM) based algorithm for computing an efficient PSNE solution that could approach a desired Pareto-optimal outcome yielding the highest social welfare among all the attainable equilibria in the long run. Our proposed learning procedure follows the regret minimization framework but extends it in three major ways: (1) agents use global, instead of local, utility for calculating regrets, (2) each agent maintains a small and diminishing exploration probability in order to explore various PSNEs, and (3) agents stay with the actions that achieve the best global utility thus far, regardless of regrets. We prove that these three extensions enable the algorithm to select the stable social optimum equilibrium instead of converging to an arbitrary or cyclic equilibrium as in the conventional RM approach. We demonstrate the effectiveness of our approach through a set of applications in multi-agent distributed control, including a large-scale resource allocation game and a hard combinatorial task assignment problem for which no efficient (polynomial) solution exists.
△ Less
Submitted 25 July, 2023;
originally announced July 2023.
-
Generative AI Perceptions: A Survey to Measure the Perceptions of Faculty, Staff, and Students on Generative AI Tools in Academia
Authors:
Sara Amani,
Lance White,
Trini Balart,
Laksha Arora,
Kristi J. Shryock,
Kelly Brumbelow,
Karan L. Watson
Abstract:
ChatGPT is a natural language processing tool that can engage in human-like conversations and generate coherent and contextually relevant responses to various prompts. ChatGPT is capable of understanding natural text that is input by a user and generating appropriate responses in various forms. This tool represents a major step in how humans are interacting with technology. This paper specifically…
▽ More
ChatGPT is a natural language processing tool that can engage in human-like conversations and generate coherent and contextually relevant responses to various prompts. ChatGPT is capable of understanding natural text that is input by a user and generating appropriate responses in various forms. This tool represents a major step in how humans are interacting with technology. This paper specifically focuses on how ChatGPT is revolutionizing the realm of engineering education and the relationship between technology, students, and faculty and staff. Because this tool is quickly changing and improving with the potential for even greater future capability, it is a critical time to collect pertinent data. A survey was created to measure the effects of ChatGPT on students, faculty, and staff. This survey is shared as a Texas A&M University technical report to allow other universities and entities to use this survey and measure the effects elsewhere.
△ Less
Submitted 21 April, 2023;
originally announced April 2023.
-
A Generative Adversarial Network for Climate Tipping Point Discovery (TIP-GAN)
Authors:
Jennifer Sleeman,
David Chung,
Anand Gnanadesikan,
Jay Brett,
Yannis Kevrekidis,
Marisa Hughes,
Thomas Haine,
Marie-Aude Pradal,
Renske Gelderloos,
Chace Ashcraft,
Caroline Tang,
Anshu Saksena,
Larry White
Abstract:
We propose a new Tipping Point Generative Adversarial Network (TIP-GAN) for better characterizing potential climate tipping points in Earth system models. We describe an adversarial game to explore the parameter space of these models, detect upcoming tipping points, and discover the drivers of tipping points. In this setup, a set of generators learn to construct model configurations that will invo…
▽ More
We propose a new Tipping Point Generative Adversarial Network (TIP-GAN) for better characterizing potential climate tipping points in Earth system models. We describe an adversarial game to explore the parameter space of these models, detect upcoming tipping points, and discover the drivers of tipping points. In this setup, a set of generators learn to construct model configurations that will invoke a climate tipping point. The discriminator learns to identify which generators are generating each model configuration and whether a given configuration will lead to a tipping point. The discriminator is trained using an oracle (a surrogate climate model) to test if a generated model configuration leads to a tipping point or not. We demonstrate the application of this GAN to invoke the collapse of the Atlantic Meridional Overturning Circulation (AMOC). We share experimental results of modifying the loss functions and the number of generators to exploit the area of uncertainty in model state space near a climate tipping point. In addition, we show that our trained discriminator can predict AMOC collapse with a high degree of accuracy without the use of the oracle. This approach could generalize to other tipping points, and could augment climate modeling research by directing users interested in studying tipping points to parameter sets likely to induce said tipping points in their computationally intensive climate models.
△ Less
Submitted 16 February, 2023;
originally announced February 2023.
-
Using Artificial Intelligence to aid Scientific Discovery of Climate Tipping Points
Authors:
Jennifer Sleeman,
David Chung,
Chace Ashcraft,
Jay Brett,
Anand Gnanadesikan,
Yannis Kevrekidis,
Marisa Hughes,
Thomas Haine,
Marie-Aude Pradal,
Renske Gelderloos,
Caroline Tang,
Anshu Saksena,
Larry White
Abstract:
We propose a hybrid Artificial Intelligence (AI) climate modeling approach that enables climate modelers in scientific discovery using a climate-targeted simulation methodology based on a novel combination of deep neural networks and mathematical methods for modeling dynamical systems. The simulations are grounded by a neuro-symbolic language that both enables question answering of what is learned…
▽ More
We propose a hybrid Artificial Intelligence (AI) climate modeling approach that enables climate modelers in scientific discovery using a climate-targeted simulation methodology based on a novel combination of deep neural networks and mathematical methods for modeling dynamical systems. The simulations are grounded by a neuro-symbolic language that both enables question answering of what is learned by the AI methods and provides a means of explainability. We describe how this methodology can be applied to the discovery of climate tipping points and, in particular, the collapse of the Atlantic Meridional Overturning Circulation (AMOC). We show how this methodology is able to predict AMOC collapse with a high degree of accuracy using a surrogate climate model for ocean interaction. We also show preliminary results of neuro-symbolic method performance when translating between natural language questions and symbolically learned representations. Our AI methodology shows promising early results, potentially enabling faster climate tipping point related research that would otherwise be computationally infeasible.
△ Less
Submitted 14 February, 2023;
originally announced February 2023.
-
Plastic Contaminant Detection in Aerial Imagery of Cotton Fields with Deep Learning
Authors:
Pappu Kumar Yadav,
J. Alex Thomasson,
Robert G. Hardin,
Stephen W. Searcy,
Ulisses Braga-Neto,
Sorin C. Popescu,
Roberto Rodriguez,
Daniel E Martin,
Juan Enciso,
Karem Meza,
Emma L. White
Abstract:
Plastic shopping bags that get carried away from the side of roads and tangled on cotton plants can end up at cotton gins if not removed before the harvest. Such bags may not only cause problem in the ginning process but might also get embodied in cotton fibers reducing its quality and marketable value. Therefore, it is required to detect, locate, and remove the bags before cotton is harvested. Ma…
▽ More
Plastic shopping bags that get carried away from the side of roads and tangled on cotton plants can end up at cotton gins if not removed before the harvest. Such bags may not only cause problem in the ginning process but might also get embodied in cotton fibers reducing its quality and marketable value. Therefore, it is required to detect, locate, and remove the bags before cotton is harvested. Manually detecting and locating these bags in cotton fields is labor intensive, time-consuming and a costly process. To solve these challenges, we present application of four variants of YOLOv5 (YOLOv5s, YOLOv5m, YOLOv5l and YOLOv5x) for detecting plastic shopping bags using Unmanned Aircraft Systems (UAS)-acquired RGB (Red, Green, and Blue) images. We also show fixed effect model tests of color of plastic bags as well as YOLOv5-variant on average precision (AP), mean average precision (mAP@50) and accuracy. In addition, we also demonstrate the effect of height of plastic bags on the detection accuracy. It was found that color of bags had significant effect (p < 0.001) on accuracy across all the four variants while it did not show any significant effect on the AP with YOLOv5m (p = 0.10) and YOLOv5x (p = 0.35) at 95% confidence level. Similarly, YOLOv5-variant did not show any significant effect on the AP (p = 0.11) and accuracy (p = 0.73) of white bags, but it had significant effects on the AP (p = 0.03) and accuracy (p = 0.02) of brown bags including on the mAP@50 (p = 0.01) and inference speed (p < 0.0001). Additionally, height of plastic bags had significant effect (p < 0.0001) on overall detection accuracy. The findings reported in this paper can be useful in speeding up removal of plastic bags from cotton fields before harvest and thereby reducing the amount of contaminants that end up at cotton gins.
△ Less
Submitted 14 December, 2022;
originally announced December 2022.
-
AbstractDifferentiation.jl: Backend-Agnostic Differentiable Programming in Julia
Authors:
Frank Schäfer,
Mohamed Tarek,
Lyndon White,
Chris Rackauckas
Abstract:
No single Automatic Differentiation (AD) system is the optimal choice for all problems. This means informed selection of an AD system and combinations can be a problem-specific variable that can greatly impact performance. In the Julia programming language, the major AD systems target the same input and thus in theory can compose. Hitherto, switching between AD packages in the Julia Language requi…
▽ More
No single Automatic Differentiation (AD) system is the optimal choice for all problems. This means informed selection of an AD system and combinations can be a problem-specific variable that can greatly impact performance. In the Julia programming language, the major AD systems target the same input and thus in theory can compose. Hitherto, switching between AD packages in the Julia Language required end-users to familiarize themselves with the user-facing API of the respective packages. Furthermore, implementing a new, usable AD package required AD package developers to write boilerplate code to define convenience API functions for end-users. As a response to these issues, we present AbstractDifferentiation.jl for the automatized generation of an extensive, unified, user-facing API for any AD package. By splitting the complexity between AD users and AD developers, AD package developers only need to implement one or two primitive definitions to support various utilities for AD users like Jacobians, Hessians and lazy product operators from native primitives such as pullbacks or pushforwards, thus removing tedious -- but so far inevitable -- boilerplate code, and enabling the easy switching and composing between AD implementations for end-users.
△ Less
Submitted 4 February, 2022; v1 submitted 25 September, 2021;
originally announced September 2021.
-
Retrofitting Effect Handlers onto OCaml
Authors:
KC Sivaramakrishnan,
Stephen Dolan,
Leo White,
Tom Kelly,
Sadiq Jaffer,
Anil Madhavapeddy
Abstract:
Effect handlers have been gathering momentum as a mechanism for modular programming with user-defined effects. Effect handlers allow for non-local control flow mechanisms such as generators, async/await, lightweight threads and coroutines to be composably expressed. We present a design and evaluate a full-fledged efficient implementation of effect handlers for OCaml, an industrial-strength multi-p…
▽ More
Effect handlers have been gathering momentum as a mechanism for modular programming with user-defined effects. Effect handlers allow for non-local control flow mechanisms such as generators, async/await, lightweight threads and coroutines to be composably expressed. We present a design and evaluate a full-fledged efficient implementation of effect handlers for OCaml, an industrial-strength multi-paradigm programming language. Our implementation strives to maintain the backwards compatibility and performance profile of existing OCaml code. Retrofitting effect handlers onto OCaml is challenging since OCaml does not currently have any non-local control flow mechanisms other than exceptions. Our implementation of effect handlers for OCaml: (i) imposes a mean 1% overhead on a comprehensive macro benchmark suite that does not use effect handlers; (ii) remains compatible with program analysis tools that inspect the stack; and (iii) is efficient for new code that makes use of effect handlers.
△ Less
Submitted 1 April, 2021;
originally announced April 2021.
-
WEmbSim: A Simple yet Effective Metric for Image Captioning
Authors:
Naeha Sharif,
Lyndon White,
Mohammed Bennamoun,
Wei Liu,
Syed Afaq Ali Shah
Abstract:
The area of automatic image caption evaluation is still undergoing intensive research to address the needs of generating captions which can meet adequacy and fluency requirements. Based on our past attempts at developing highly sophisticated learning-based metrics, we have discovered that a simple cosine similarity measure using the Mean of Word Embeddings(MOWE) of captions can actually achieve a…
▽ More
The area of automatic image caption evaluation is still undergoing intensive research to address the needs of generating captions which can meet adequacy and fluency requirements. Based on our past attempts at developing highly sophisticated learning-based metrics, we have discovered that a simple cosine similarity measure using the Mean of Word Embeddings(MOWE) of captions can actually achieve a surprisingly high performance on unsupervised caption evaluation. This inspires our proposed work on an effective metric WEmbSim, which beats complex measures such as SPICE, CIDEr and WMD at system-level correlation with human judgments. Moreover, it also achieves the best accuracy at matching human consensus scores for caption pairs, against commonly used unsupervised methods. Therefore, we believe that WEmbSim sets a new baseline for any complex metric to be justified.
△ Less
Submitted 24 December, 2020;
originally announced December 2020.
-
LCEval: Learned Composite Metric for Caption Evaluation
Authors:
Naeha Sharif,
Lyndon White,
Mohammed Bennamoun,
Wei Liu,
Syed Afaq Ali Shah
Abstract:
Automatic evaluation metrics hold a fundamental importance in the development and fine-grained analysis of captioning systems. While current evaluation metrics tend to achieve an acceptable correlation with human judgements at the system level, they fail to do so at the caption level. In this work, we propose a neural network-based learned metric to improve the caption-level caption evaluation. To…
▽ More
Automatic evaluation metrics hold a fundamental importance in the development and fine-grained analysis of captioning systems. While current evaluation metrics tend to achieve an acceptable correlation with human judgements at the system level, they fail to do so at the caption level. In this work, we propose a neural network-based learned metric to improve the caption-level caption evaluation. To get a deeper insight into the parameters which impact a learned metrics performance, this paper investigates the relationship between different linguistic features and the caption-level correlation of the learned metrics. We also compare metrics trained with different training examples to measure the variations in their evaluation. Moreover, we perform a robustness analysis, which highlights the sensitivity of learned and handcrafted metrics to various sentence perturbations. Our empirical analysis shows that our proposed metric not only outperforms the existing metrics in terms of caption-level correlation but it also shows a strong system-level correlation against human assessments.
△ Less
Submitted 24 December, 2020;
originally announced December 2020.
-
Retrofitting Parallelism onto OCaml
Authors:
KC Sivaramakrishnan,
Stephen Dolan,
Leo White,
Sadiq Jaffer,
Tom Kelly,
Anmol Sahoo,
Sudha Parimala,
Atul Dhiman,
Anil Madhavapeddy
Abstract:
OCaml is an industrial-strength, multi-paradigm programming language, widely used in industry and academia. OCaml is also one of the few modern managed system programming languages to lack support for shared memory parallel programming. This paper describes the design, a full-fledged implementation and evaluation of a mostly-concurrent garbage collector (GC) for the multicore extension of the OCam…
▽ More
OCaml is an industrial-strength, multi-paradigm programming language, widely used in industry and academia. OCaml is also one of the few modern managed system programming languages to lack support for shared memory parallel programming. This paper describes the design, a full-fledged implementation and evaluation of a mostly-concurrent garbage collector (GC) for the multicore extension of the OCaml programming language. Given that we propose to add parallelism to a widely used programming language with millions of lines of existing code, we face the challenge of maintaining backwards compatibility--not just in terms of the language features but also the performance of single-threaded code running with the new GC. To this end, the paper presents a series of novel techniques and demonstrates that the new GC strikes a balance between performance and feature backwards compatibility for sequential programs and scales admirably on modern multicore processors.
△ Less
Submitted 2 July, 2020; v1 submitted 24 April, 2020;
originally announced April 2020.
-
Robust Archives Maximize Scientific Accessibility
Authors:
J. E. G. Peek,
Vandana Desai,
Richard L. White,
Raffaele D'Abrusco,
Joseph M. Mazzarella,
Carolyn Grant,
Jenny L. Novacescu,
Elena Scire,
Sherry Winkelman
Abstract:
We present a bibliographic analysis of Chandra, Hubble, and Spitzer publications. We find (a) archival data are used in >60% of the publication output and (b) archives for these missions enable a much broader set of institutions and countries to scientifically use data from these missions. Specifically, we find that authors from institutions that have published few papers from a given mission publ…
▽ More
We present a bibliographic analysis of Chandra, Hubble, and Spitzer publications. We find (a) archival data are used in >60% of the publication output and (b) archives for these missions enable a much broader set of institutions and countries to scientifically use data from these missions. Specifically, we find that authors from institutions that have published few papers from a given mission publish 2/3 archival publications, while those with many publications typically have 1/3 archival publications. We also show that countries with lower GDP per capita overwhelmingly produce archival publications, while countries with higher GDP per capital produce guest observer and archival publications in equal amounts. We argue that robust archives are thus not only critical for the scientific productivity of mission data, but also the scientific accessibility of mission data. We argue that the astronomical community should support archives to maximize the overall scientific societal impact of astronomy, and represent an excellent investment in astronomy's future.
△ Less
Submitted 14 July, 2019;
originally announced July 2019.
-
DiffEqFlux.jl - A Julia Library for Neural Differential Equations
Authors:
Chris Rackauckas,
Mike Innes,
Yingbo Ma,
Jesse Bettencourt,
Lyndon White,
Vaibhav Dixit
Abstract:
DiffEqFlux.jl is a library for fusing neural networks and differential equations. In this work we describe differential equations from the viewpoint of data science and discuss the complementary nature between machine learning models and differential equations. We demonstrate the ability to incorporate DifferentialEquations.jl-defined differential equation problems into a Flux-defined neural netwo…
▽ More
DiffEqFlux.jl is a library for fusing neural networks and differential equations. In this work we describe differential equations from the viewpoint of data science and discuss the complementary nature between machine learning models and differential equations. We demonstrate the ability to incorporate DifferentialEquations.jl-defined differential equation problems into a Flux-defined neural network, and vice versa. The advantages of being able to use the entire DifferentialEquations.jl suite for this purpose is demonstrated by counter examples where simple integration strategies fail, but the sophisticated integration strategies provided by the DifferentialEquations.jl library succeed. This is followed by a demonstration of delay differential equations and stochastic differential equations inside of neural networks. We show high-level functionality for defining neural ordinary differential equations (neural networks embedded into the differential equation) and describe the extra models in the Flux model zoo which includes neural stochastic differential equations. We conclude by discussing the various adjoint methods used for backpropogation of the differential equation solvers. DiffEqFlux.jl is an important contribution to the area, as it allows the full weight of the differential equation solvers developed from decades of research in the scientific computing field to be readily applied to the challenges posed by machine learning and data science.
△ Less
Submitted 6 February, 2019;
originally announced February 2019.
-
Critical initialisation in continuous approximations of binary neural networks
Authors:
George Stamatescu,
Federica Gerace,
Carlo Lucibello,
Ian Fuss,
Langford B. White
Abstract:
The training of stochastic neural network models with binary ($\pm1$) weights and activations via continuous surrogate networks is investigated. We derive new surrogates using a novel derivation based on writing the stochastic neural network as a Markov chain. This derivation also encompasses existing variants of the surrogates presented in the literature. Following this, we theoretically study th…
▽ More
The training of stochastic neural network models with binary ($\pm1$) weights and activations via continuous surrogate networks is investigated. We derive new surrogates using a novel derivation based on writing the stochastic neural network as a Markov chain. This derivation also encompasses existing variants of the surrogates presented in the literature. Following this, we theoretically study the surrogates at initialisation. We derive, using mean field theory, a set of scalar equations describing how input signals propagate through the randomly initialised networks. The equations reveal whether so-called critical initialisations exist for each surrogate network, where the network can be trained to arbitrary depth. Moreover, we predict theoretically and confirm numerically, that common weight initialisation schemes used in standard continuous networks, when applied to the mean values of the stochastic binary weights, yield poor training performance. This study shows that, contrary to common intuition, the means of the stochastic binary weights should be initialised close to $\pm 1$, for deeper networks to be trainable.
△ Less
Submitted 9 April, 2020; v1 submitted 31 January, 2019;
originally announced February 2019.
-
DataDeps.jl: Repeatable Data Setup for Replicable Data Science
Authors:
Lyndon White,
Roberto Togneri,
Wei Liu,
Mohammed Bennamoun
Abstract:
We present DataDeps.jl: a julia package for the reproducible handling of static datasets to enhance the repeatability of scripts used in the data and computational sciences. It is used to automate the data setup part of running software which accompanies a paper to replicate a result. This step is commonly done manually, which expends time and allows for confusion. This functionality is also usefu…
▽ More
We present DataDeps.jl: a julia package for the reproducible handling of static datasets to enhance the repeatability of scripts used in the data and computational sciences. It is used to automate the data setup part of running software which accompanies a paper to replicate a result. This step is commonly done manually, which expends time and allows for confusion. This functionality is also useful for other packages which require data to function (e.g. a trained machine learning based model). DataDeps.jl simplifies extending research software by automatically managing the dependencies and makes it easier to run another author's code, thus enhancing the reproducibility of data science research.
△ Less
Submitted 3 August, 2018;
originally announced August 2018.
-
Learning of Colors from Color Names: Distribution and Point Estimation
Authors:
Lyndon White,
Roberto Togneri,
Wei Liu,
Mohammed Bennamoun
Abstract:
Color names are often made up of multiple words. As a task in natural language understanding we investigate in depth the capacity of neural networks based on sums of word embeddings (SOWE), recurrence (LSTM and GRU based RNNs) and convolution (CNN), to estimate colors from sequences of terms. We consider both point and distribution estimates of color. We argue that the latter has a particular valu…
▽ More
Color names are often made up of multiple words. As a task in natural language understanding we investigate in depth the capacity of neural networks based on sums of word embeddings (SOWE), recurrence (LSTM and GRU based RNNs) and convolution (CNN), to estimate colors from sequences of terms. We consider both point and distribution estimates of color. We argue that the latter has a particular value as there is no clear agreement between people as to what a particular color describes -- different people have a different idea of what it means to be ``very dark orange'', for example. Surprisingly, despite it's simplicity, the sum of word embeddings generally performs the best on almost all evaluations.
△ Less
Submitted 10 January, 2020; v1 submitted 27 September, 2017;
originally announced September 2017.
-
Track Extraction with Hidden Reciprocal Chain Models
Authors:
George Stamatescu,
Langford B White,
Riley Bruce-Doust
Abstract:
This paper develops Bayesian track extraction algorithms for targets modelled as hidden reciprocal chains (HRC). HRC are a class of finite-state random process models that generalise the familiar hidden Markov chains (HMC). HRC are able to model the "intention" of a target to proceed from a given origin to a destination, behaviour which cannot be properly captured by a HMC. While Bayesian estimati…
▽ More
This paper develops Bayesian track extraction algorithms for targets modelled as hidden reciprocal chains (HRC). HRC are a class of finite-state random process models that generalise the familiar hidden Markov chains (HMC). HRC are able to model the "intention" of a target to proceed from a given origin to a destination, behaviour which cannot be properly captured by a HMC. While Bayesian estimation problems for HRC have previously been studied, this paper focusses principally on the problem of track extraction, of which the primary task is confirming target existence in a set of detections obtained from thresholding sensor measurements. Simulation examples are presented which show that the additional model information contained in a HRC improves detection performance when compared to HMC models.
△ Less
Submitted 13 May, 2016;
originally announced May 2016.
-
Modular implicits
Authors:
Leo White,
Frédéric Bour,
Jeremy Yallop
Abstract:
We present modular implicits, an extension to the OCaml language for ad-hoc polymorphism inspired by Scala implicits and modular type classes. Modular implicits are based on type-directed implicit module parameters, and elaborate straightforwardly into OCaml's first-class functors. Basing the design on OCaml's modules leads to a system that naturally supports many features from other languages wit…
▽ More
We present modular implicits, an extension to the OCaml language for ad-hoc polymorphism inspired by Scala implicits and modular type classes. Modular implicits are based on type-directed implicit module parameters, and elaborate straightforwardly into OCaml's first-class functors. Basing the design on OCaml's modules leads to a system that naturally supports many features from other languages with systematic ad-hoc overloading, including inheritance, instance constraints, constructor classes and associated types.
△ Less
Submitted 6 December, 2015;
originally announced December 2015.
-
Throwing Out the Baby with the Bathwater: The Undesirable Effects of National Research Assessment Exercises on Research
Authors:
John Mingers,
Leroy White
Abstract:
The evaluation of the quality of research at a national level has become increasingly common. The UK has been at the forefront of this trend having undertaken many assessments since 1986, the latest being the Research Excellence Framework in 2014. The argument of this paper is that, whatever the intended results in terms of evaluating and improving research, there have been many, presumably uninte…
▽ More
The evaluation of the quality of research at a national level has become increasingly common. The UK has been at the forefront of this trend having undertaken many assessments since 1986, the latest being the Research Excellence Framework in 2014. The argument of this paper is that, whatever the intended results in terms of evaluating and improving research, there have been many, presumably unintended, results that are highly undesirable for research and the university community more generally. We situate our analysis using Bourdieu's theory of cultural reproduction and then focus on the peculiarities of the 2008 RAE and the 2014 REF the rules of which allowed for, and indeed encouraged, significant game-playing on the part of striving universities. We conclude with practical recommendations to maintain the general intention of research assessment without the undesirable side-effects.
△ Less
Submitted 2 February, 2015;
originally announced February 2015.
-
A Tiled-Table Convention for Compressing FITS Binary Tables
Authors:
William Pence,
Rob Seaman,
Richard L. White
Abstract:
This document describes a convention for compressing FITS binary tables that is modeled after the FITS tiled-image compression method (White et al. 2009) that has been in use for about a decade. The input table is first optionally subdivided into tiles, each containing an equal number of rows, then every column of data within each tile is compressed and stored as a variable-length array of bytes i…
▽ More
This document describes a convention for compressing FITS binary tables that is modeled after the FITS tiled-image compression method (White et al. 2009) that has been in use for about a decade. The input table is first optionally subdivided into tiles, each containing an equal number of rows, then every column of data within each tile is compressed and stored as a variable-length array of bytes in the output FITS binary table. All the header keywords from the input table are copied to the header of the output table and remain uncompressed for efficient access. The output compressed table contains the same number and order of columns as in the input uncompressed binary table. There is one row in the output table corresponding to each tile of rows in the input table. In principle, each column of data can be compressed using a different algorithm that is optimized for the type of data within that column, however in the prototype implementation described here, the gzip algorithm is used to compress every column.
△ Less
Submitted 5 January, 2012;
originally announced January 2012.