-
Introducing v0.5 of the AI Safety Benchmark from MLCommons
Authors:
Bertie Vidgen,
Adarsh Agrawal,
Ahmed M. Ahmed,
Victor Akinwande,
Namir Al-Nuaimi,
Najla Alfaraj,
Elie Alhajjar,
Lora Aroyo,
Trupti Bavalatti,
Max Bartolo,
Borhane Blili-Hamelin,
Kurt Bollacker,
Rishi Bomassani,
Marisa Ferrara Boston,
Siméon Campos,
Kal Chakra,
Canyu Chen,
Cody Coleman,
Zacharie Delpierre Coudert,
Leon Derczynski,
Debojyoti Dutta,
Ian Eisenberg,
James Ezick,
Heather Frase,
Brian Fuller
, et al. (75 additional authors not shown)
Abstract:
This paper introduces v0.5 of the AI Safety Benchmark, which has been created by the MLCommons AI Safety Working Group. The AI Safety Benchmark has been designed to assess the safety risks of AI systems that use chat-tuned language models. We introduce a principled approach to specifying and constructing the benchmark, which for v0.5 covers only a single use case (an adult chatting to a general-pu…
▽ More
This paper introduces v0.5 of the AI Safety Benchmark, which has been created by the MLCommons AI Safety Working Group. The AI Safety Benchmark has been designed to assess the safety risks of AI systems that use chat-tuned language models. We introduce a principled approach to specifying and constructing the benchmark, which for v0.5 covers only a single use case (an adult chatting to a general-purpose assistant in English), and a limited set of personas (i.e., typical users, malicious users, and vulnerable users). We created a new taxonomy of 13 hazard categories, of which 7 have tests in the v0.5 benchmark. We plan to release version 1.0 of the AI Safety Benchmark by the end of 2024. The v1.0 benchmark will provide meaningful insights into the safety of AI systems. However, the v0.5 benchmark should not be used to assess the safety of AI systems. We have sought to fully document the limitations, flaws, and challenges of v0.5. This release of v0.5 of the AI Safety Benchmark includes (1) a principled approach to specifying and constructing the benchmark, which comprises use cases, types of systems under test (SUTs), language and context, personas, tests, and test items; (2) a taxonomy of 13 hazard categories with definitions and subcategories; (3) tests for seven of the hazard categories, each comprising a unique set of test items, i.e., prompts. There are 43,090 test items in total, which we created with templates; (4) a grading system for AI systems against the benchmark; (5) an openly available platform, and downloadable tool, called ModelBench that can be used to evaluate the safety of AI systems on the benchmark; (6) an example evaluation report which benchmarks the performance of over a dozen openly available chat-tuned language models; (7) a test specification for the benchmark.
△ Less
Submitted 13 May, 2024; v1 submitted 18 April, 2024;
originally announced April 2024.
-
On locally finite ordered rooted trees and their rooted subtrees
Authors:
Geir Agnarsson,
Elie Alhajjar,
Aleyah Dawkins
Abstract:
In this article we compare the known dynamical polynomial time algorithm for the game-over attack strategy, to that of the brute force approach; of checking all the ordered rooted subtrees of a given tree that represents a given computer network. Our approach is purely enumerative and combinatorial in nature. We first revisit known results about a doubly exponential sequence and generalize them. W…
▽ More
In this article we compare the known dynamical polynomial time algorithm for the game-over attack strategy, to that of the brute force approach; of checking all the ordered rooted subtrees of a given tree that represents a given computer network. Our approach is purely enumerative and combinatorial in nature. We first revisit known results about a doubly exponential sequence and generalize them. We then consider both finite and locally finite ordered rooted trees (LFOR-trees), and the class of their finite ordered rooted subtrees of bounded height, describing completely the LFOR-trees with no leaves where the number of ordered rooted subtrees of height at most $h$ are bounded by a polynomial in $h$. We finally consider general LFOR-trees where each level can have leaves and determine conditions for the number of ordered rooted subtrees of height at most $h$ to be bounded by a polynomial in $h$.
△ Less
Submitted 18 December, 2023;
originally announced December 2023.
-
Efficient quantum loading of probability distributions through Feynman propagators
Authors:
Elie Alhajjar,
Jesse Geneson,
Anupam Prakash,
Nicolas Robles
Abstract:
We present quantum algorithms for the loading of probability distributions using Hamiltonian simulation for one dimensional Hamiltonians of the form ${\hat H}= Δ+ V(x) \mathbb{I}$. We consider the potentials $V(x)$ for which the Feynman propagator is known to have an analytically closed form and utilize these Hamiltonians to load probability distributions including the normal, Laplace and Maxwell-…
▽ More
We present quantum algorithms for the loading of probability distributions using Hamiltonian simulation for one dimensional Hamiltonians of the form ${\hat H}= Δ+ V(x) \mathbb{I}$. We consider the potentials $V(x)$ for which the Feynman propagator is known to have an analytically closed form and utilize these Hamiltonians to load probability distributions including the normal, Laplace and Maxwell-Boltzmann into quantum states. We also propose a variational method for probability distribution loading based on constructing a coarse approximation to the distribution in the form of a `ladder state' and then projecting onto the ground state of a Hamiltonian chosen to have the desired probability distribution as ground state. These methods extend the suite of techniques available for the loading of probability distributions, and are more efficient than general purpose data loading methods used in quantum machine learning.
△ Less
Submitted 28 November, 2023; v1 submitted 22 November, 2023;
originally announced November 2023.
-
A Machine Learning Pressure Emulator for Hydrogen Embrittlement
Authors:
Minh Triet Chau,
João Lucas de Sousa Almeida,
Elie Alhajjar,
Alberto Costa Nogueira Junior
Abstract:
A recent alternative for hydrogen transportation as a mixture with natural gas is blending it into natural gas pipelines. However, hydrogen embrittlement of material is a major concern for scientists and gas installation designers to avoid process failures. In this paper, we propose a physics-informed machine learning model to predict the gas pressure on the pipes' inner wall. Despite its high-fid…
▽ More
A recent alternative for hydrogen transportation as a mixture with natural gas is blending it into natural gas pipelines. However, hydrogen embrittlement of material is a major concern for scientists and gas installation designers to avoid process failures. In this paper, we propose a physics-informed machine learning model to predict the gas pressure on the pipes' inner wall. Despite its high-fidelity results, the current PDE-based simulators are time- and computationally-demanding. Using simulation data, we train an ML model to predict the pressure on the pipelines' inner walls, which is a first step for pipeline system surveillance. We found that the physics-based method outperformed the purely data-driven method and satisfy the physical constraints of the gas flow system.
△ Less
Submitted 22 June, 2023;
originally announced June 2023.
-
Novelty Detection in Network Traffic: Using Survival Analysis for Feature Identification
Authors:
Taylor Bradley,
Elie Alhajjar,
Nathaniel Bastian
Abstract:
Intrusion Detection Systems are an important component of many organizations' cyber defense and resiliency strategies. However, one downside of these systems is their reliance on known attack signatures for detection of malicious network events. When it comes to unknown attack types and zero-day exploits, modern Intrusion Detection Systems often fall short. In this paper, we introduce an unconvent…
▽ More
Intrusion Detection Systems are an important component of many organizations' cyber defense and resiliency strategies. However, one downside of these systems is their reliance on known attack signatures for detection of malicious network events. When it comes to unknown attack types and zero-day exploits, modern Intrusion Detection Systems often fall short. In this paper, we introduce an unconventional approach to identifying network traffic features that influence novelty detection based on survival analysis techniques. Specifically, we combine several Cox proportional hazards models and implement Kaplan-Meier estimates to predict the probability that a classifier identifies novelty after the injection of an unknown network attack at any given time. The proposed model is successful at pinpointing PSH Flag Count, ACK Flag Count, URG Flag Count, and Down/Up Ratio as the main features to impact novelty detection via Random Forest, Bayesian Ridge, and Linear Support Vector Regression classifiers.
△ Less
Submitted 15 January, 2023;
originally announced January 2023.
-
Cubature Kalman Filter Based Training of Hybrid Differential Equation Recurrent Neural Network Physiological Dynamic Models
Authors:
Ahmet Demirkaya,
Tales Imbiriba,
Kyle Lockwood,
Sumientra Rampersad,
Elie Alhajjar,
Giovanna Guidoboni,
Zachary Danziger,
Deniz Erdogmus
Abstract:
Modeling biological dynamical systems is challenging due to the interdependence of different system components, some of which are not fully understood. To fill existing gaps in our ability to mechanistically model physiological systems, we propose to combine neural networks with physics-based models. Specifically, we demonstrate how we can approximate missing ordinary differential equations (ODEs)…
▽ More
Modeling biological dynamical systems is challenging due to the interdependence of different system components, some of which are not fully understood. To fill existing gaps in our ability to mechanistically model physiological systems, we propose to combine neural networks with physics-based models. Specifically, we demonstrate how we can approximate missing ordinary differential equations (ODEs) coupled with known ODEs using Bayesian filtering techniques to train the model parameters and simultaneously estimate dynamic state variables. As a study case we leverage a well-understood model for blood circulation in the human retina and replace one of its core ODEs with a neural network approximation, representing the case where we have incomplete knowledge of the physiological state dynamics. Results demonstrate that state dynamics corresponding to the missing ODEs can be approximated well using a neural network trained using a recursive Bayesian filtering approach in a fashion coupled with the known state dynamic differential equations. This demonstrates that dynamics and impact of missing state variables can be captured through joint state estimation and model parameter estimation within a recursive Bayesian state estimation (RBSE) framework. Results also indicate that this RBSE approach to training the NN parameters yields better outcomes (measurement/state estimation accuracy) than training the neural network with backpropagation through time in the same setting.
△ Less
Submitted 12 October, 2021;
originally announced October 2021.
-
Maximally entangled correlation sets
Authors:
Elie Alhajjar,
Travis Russell
Abstract:
We study the set of quantum correlations generated by actions on maximally entangled states. We show that such correlations are dense in their own convex hull. As a consequence, we show that these correlations are dense in the set of synchronous quantum correlations. We introduce the concept of corners of correlation sets and show that every local or nonsignalling correlation can be realized as th…
▽ More
We study the set of quantum correlations generated by actions on maximally entangled states. We show that such correlations are dense in their own convex hull. As a consequence, we show that these correlations are dense in the set of synchronous quantum correlations. We introduce the concept of corners of correlation sets and show that every local or nonsignalling correlation can be realized as the corner of a synchronous local or nonsignalling correlation. We provide partial results for other correlation sets.
△ Less
Submitted 2 June, 2020;
originally announced June 2020.
-
Adversarial Machine Learning in Network Intrusion Detection Systems
Authors:
Elie Alhajjar,
Paul Maxwell,
Nathaniel D. Bastian
Abstract:
Adversarial examples are inputs to a machine learning system intentionally crafted by an attacker to fool the model into producing an incorrect output. These examples have achieved a great deal of success in several domains such as image recognition, speech recognition and spam detection. In this paper, we study the nature of the adversarial problem in Network Intrusion Detection Systems (NIDS). W…
▽ More
Adversarial examples are inputs to a machine learning system intentionally crafted by an attacker to fool the model into producing an incorrect output. These examples have achieved a great deal of success in several domains such as image recognition, speech recognition and spam detection. In this paper, we study the nature of the adversarial problem in Network Intrusion Detection Systems (NIDS). We focus on the attack perspective, which includes techniques to generate adversarial examples capable of evading a variety of machine learning models. More specifically, we explore the use of evolutionary computation (particle swarm optimization and genetic algorithm) and deep learning (generative adversarial networks) as tools for adversarial example generation. To assess the performance of these algorithms in evading a NIDS, we apply them to two publicly available data sets, namely the NSL-KDD and UNSW-NB15, and we contrast them to a baseline perturbation method: Monte Carlo simulation. The results show that our adversarial example generation techniques cause high misclassification rates in eleven different machine learning models, along with a voting classifier. Our work highlights the vulnerability of machine learning based NIDS in the face of adversarial perturbation.
△ Less
Submitted 23 April, 2020;
originally announced April 2020.
-
Ehrhart positivity and Demazure characters
Authors:
Per Alexandersson,
Elie Alhajjar
Abstract:
Demazure characters, also known as key polynomials, generalize the classical Schur polynomials. In particular, when all variables are set equal to $1$, these polynomials count the number of integer points in a certain class of Gelfand--Tsetlin polytopes. This property highlights the interaction between the corresponding polyhedral and combinatorial structures via Ehrhart theory. In this paper, we…
▽ More
Demazure characters, also known as key polynomials, generalize the classical Schur polynomials. In particular, when all variables are set equal to $1$, these polynomials count the number of integer points in a certain class of Gelfand--Tsetlin polytopes. This property highlights the interaction between the corresponding polyhedral and combinatorial structures via Ehrhart theory. In this paper, we give an overview of results concerning the interplay between the geometry of Gelfand-Tsetlin polytopes and their Ehrhart polynomials. Motivated by strong computer evidence, we propose several conjectures about the non-negativity of the coefficients of such polynomials.
△ Less
Submitted 8 December, 2018;
originally announced December 2018.
