Showing 1–50 of 316 results for author: Chan, A

Open Problems in Technical AI Governance

Authors: Anka Reuel, Ben Bucknall, Stephen Casper, Tim Fist, Lisa Soder, Onni Aarne, Lewis Hammond, Lujain Ibrahim, Alan Chan, Peter Wills, Markus Anderljung, Ben Garfinkel, Lennart Heim, Andrew Trask, Gabriel Mukobi, Rylan Schaeffer, Mauricio Baker, Sara Hooker, Irene Solaiman, Alexandra Sasha Luccioni, Nitarshan Rajkumar, Nicolas Moës, Jeffrey Ladish, Neel Guha, Jessica Newman , et al. (6 additional authors not shown)

Abstract: AI progress is creating a growing range of risks and opportunities, but it is often unclear how they should be navigated. In many cases, the barriers and uncertainties faced are at least partly technical. Technical AI governance, referring to technical analysis and tools for supporting the effective governance of AI, seeks to address such challenges. It can help to (a) identify areas where interve… ▽ More AI progress is creating a growing range of risks and opportunities, but it is often unclear how they should be navigated. In many cases, the barriers and uncertainties faced are at least partly technical. Technical AI governance, referring to technical analysis and tools for supporting the effective governance of AI, seeks to address such challenges. It can help to (a) identify areas where intervention is needed, (b) identify and assess the efficacy of potential governance actions, and (c) enhance governance options by designing mechanisms for enforcement, incentivization, or compliance. In this paper, we explain what technical AI governance is, why it is important, and present a taxonomy and incomplete catalog of its open problems. This paper is intended as a resource for technical researchers or research funders looking to contribute to AI governance. △ Less

Submitted 20 July, 2024; originally announced July 2024.

Comments: Ben Bucknall and Anka Reuel contributed equally and share the first author position

Single-mode emission by phase-delayed coupling between nano-lasers

Authors: T. V. Raziman, Anna Fischer, Riccardo Nori, Anthony Chan, Wai Kit Ng, Dhruv Saxena, Ortwin Hess, Korneel Molkens, Ivo Tanghe, Pieter Geiregat, Dries Van Thourhout, Mauricio Barahona, Riccardo Sapienza

Abstract: Near-field coupling between nanolasers enables collective high-power lasing but leads to complex spectral reshaping and multimode operation, limiting the emission brightness, spatial coherence and temporal stability. Many lasing architectures have been proposed to circumvent this limitation, based on symmetries, topology, or interference. We show that a much simpler and robust method exploiting ph… ▽ More Near-field coupling between nanolasers enables collective high-power lasing but leads to complex spectral reshaping and multimode operation, limiting the emission brightness, spatial coherence and temporal stability. Many lasing architectures have been proposed to circumvent this limitation, based on symmetries, topology, or interference. We show that a much simpler and robust method exploiting phase-delayed coupling, where light exchanged by the lasers carries a phase, can enable stable single-mode operation. Phase-delayed coupling changes the modal amplification: for pump powers close to the anyonic parity-time (PT) symmetric exceptional point, a high phase delay completely separates the mode thresholds, leading to single mode operation. This is shown by stability analysis with nonlinear coupled mode theory and stochastic differential equations for two coupled nanolasers and confirmed by realistic semi-analytical treatment of a dimer of lasing nanospheres. Finally, we extend the mode control to large arrays of nanolasers, featuring lowered thresholds and higher power. Our work promises a novel solution to engineer bright and stable single-mode lasing from nanolaser arrays with important applications in photonic chips for communication and lidars. △ Less

Submitted 4 July, 2024; originally announced July 2024.

On the 96-well plate coverglass tilt and curvature suppression in 96-camera imaging system

Authors: Antony C Chan

Abstract: The 96-eyes instrument is capable of computational extended depth of focus (eDOF) of up to +/- 30 micrometer in the phase channel, and conventional depth of field (DOF) of +/- 5 micrometer in the fluorescence channel. However, it requires minimal plate-to-plate cover glass depth variation to function. Plate depths are measured using a third-party plate scanner (Opera Phenix) grouped by plate types… ▽ More The 96-eyes instrument is capable of computational extended depth of focus (eDOF) of up to +/- 30 micrometer in the phase channel, and conventional depth of field (DOF) of +/- 5 micrometer in the fluorescence channel. However, it requires minimal plate-to-plate cover glass depth variation to function. Plate depths are measured using a third-party plate scanner (Opera Phenix) grouped by plate types (Greiner UV-Star, Cell-Star, and Eppendorf meniscus-free). The two-dimensional (2D) depth dataset is aggregated through principal component analysis to obtain the top eight dominating 2D surface deformation modes. More than 90% of the variation can be explained by the plate's absolute depth and tilt (Pitch, Gradient-Y, and Gradient-X), followed by (~= 2%) the cover glass's curvature (Curve-Y and Curve-XY). Plate-to-plate average depth and tilt variations are suppressed by a customized kinematic mount anchoring the plate's cover glass at the instrument's imaging plane. The plate's average curvature is compensated by manually aligning all 96-eyes microscope objective lenses to track the plate's surface; an one-off calibration procedure aided by the backlash-free piezo-flexure z-stage. Design validation is conducted in silico, with the proof of concept experiment conducted on the 96-eyes with new mounting bracket retrofits. △ Less

Submitted 13 March, 2024; originally announced June 2024.

IDs for AI Systems

Authors: Alan Chan, Noam Kolt, Peter Wills, Usman Anwar, Christian Schroeder de Witt, Nitarshan Rajkumar, Lewis Hammond, David Krueger, Lennart Heim, Markus Anderljung

Abstract: AI systems are increasingly pervasive, yet information needed to decide whether and how to engage with them may not exist or be accessible. A user may not be able to verify whether a system has certain safety certifications. An investigator may not know whom to investigate when a system causes an incident. It may not be clear whom to contact to shut down a malfunctioning system. Across a number of… ▽ More AI systems are increasingly pervasive, yet information needed to decide whether and how to engage with them may not exist or be accessible. A user may not be able to verify whether a system has certain safety certifications. An investigator may not know whom to investigate when a system causes an incident. It may not be clear whom to contact to shut down a malfunctioning system. Across a number of domains, IDs address analogous problems by identifying particular entities (e.g., a particular Boeing 747) and providing information about other entities of the same class (e.g., some or all Boeing 747s). We propose a framework in which IDs are ascribed to instances of AI systems (e.g., a particular chat session with Claude 3), and associated information is accessible to parties seeking to interact with that system. We characterize IDs for AI systems, provide concrete examples where IDs could be useful, argue that there could be significant demand for IDs from key actors, analyze how those actors could incentivize ID adoption, explore a potential implementation of our framework for deployers of AI systems, and highlight limitations and risks. IDs seem most warranted in settings where AI systems could have a large impact upon the world, such as in making financial transactions or contacting real humans. With further study, IDs could help to manage a world where AI systems pervade society. △ Less

Submitted 18 July, 2024; v1 submitted 17 June, 2024; originally announced June 2024.

Comments: Work-in-progress

Muharaf: Manuscripts of Handwritten Arabic Dataset for Cursive Text Recognition

Authors: Mehreen Saeed, Adrian Chan, Anupam Mijar, Joseph Moukarzel, Georges Habchi, Carlos Younes, Amin Elias, Chau-Wai Wong, Akram Khater

Abstract: We present the Manuscripts of Handwritten Arabic~(Muharaf) dataset, which is a machine learning dataset consisting of more than 1,600 historic handwritten page images transcribed by experts in archival Arabic. Each document image is accompanied by spatial polygonal coordinates of its text lines as well as basic page elements. This dataset was compiled to advance the state of the art in handwritten… ▽ More We present the Manuscripts of Handwritten Arabic~(Muharaf) dataset, which is a machine learning dataset consisting of more than 1,600 historic handwritten page images transcribed by experts in archival Arabic. Each document image is accompanied by spatial polygonal coordinates of its text lines as well as basic page elements. This dataset was compiled to advance the state of the art in handwritten text recognition (HTR), not only for Arabic manuscripts but also for cursive text in general. The Muharaf dataset includes diverse handwriting styles and a wide range of document types, including personal letters, diaries, notes, poems, church records, and legal correspondences. In this paper, we describe the data acquisition pipeline, notable dataset features, and statistics. We also provide a preliminary baseline result achieved by training convolutional neural networks using this data. △ Less

Submitted 13 June, 2024; originally announced June 2024.

CodedEvents: Optimal Point-Spread-Function Engineering for 3D-Tracking with Event Cameras

Authors: Sachin Shah, Matthew Albert Chan, Haoming Cai, Jingxi Chen, Sakshum Kulshrestha, Chahat Deep Singh, Yiannis Aloimonos, Christopher Metzler

Abstract: Point-spread-function (PSF) engineering is a well-established computational imaging technique that uses phase masks and other optical elements to embed extra information (e.g., depth) into the images captured by conventional CMOS image sensors. To date, however, PSF-engineering has not been applied to neuromorphic event cameras; a powerful new image sensing technology that responds to changes in t… ▽ More Point-spread-function (PSF) engineering is a well-established computational imaging technique that uses phase masks and other optical elements to embed extra information (e.g., depth) into the images captured by conventional CMOS image sensors. To date, however, PSF-engineering has not been applied to neuromorphic event cameras; a powerful new image sensing technology that responds to changes in the log-intensity of light. This paper establishes theoretical limits (Cramér Rao bounds) on 3D point localization and tracking with PSF-engineered event cameras. Using these bounds, we first demonstrate that existing Fisher phase masks are already near-optimal for localizing static flashing point sources (e.g., blinking fluorescent molecules). We then demonstrate that existing designs are sub-optimal for tracking moving point sources and proceed to use our theory to design optimal phase masks and binary amplitude masks for this task. To overcome the non-convexity of the design problem, we leverage novel implicit neural representation based parameterizations of the phase and amplitude masks. We demonstrate the efficacy of our designs through extensive simulations. We also validate our method with a simple prototype. △ Less

Submitted 13 June, 2024; originally announced June 2024.

Discovering Preference Optimization Algorithms with and for Large Language Models

Authors: Chris Lu, Samuel Holt, Claudio Fanconi, Alex J. Chan, Jakob Foerster, Mihaela van der Schaar, Robert Tjarko Lange

Abstract: Offline preference optimization is a key method for enhancing and controlling the quality of Large Language Model (LLM) outputs. Typically, preference optimization is approached as an offline supervised learning task using manually-crafted convex loss functions. While these methods are based on theoretical insights, they are inherently constrained by human creativity, so the large search space of… ▽ More Offline preference optimization is a key method for enhancing and controlling the quality of Large Language Model (LLM) outputs. Typically, preference optimization is approached as an offline supervised learning task using manually-crafted convex loss functions. While these methods are based on theoretical insights, they are inherently constrained by human creativity, so the large search space of possible loss functions remains under explored. We address this by performing LLM-driven objective discovery to automatically discover new state-of-the-art preference optimization algorithms without (expert) human intervention. Specifically, we iteratively prompt an LLM to propose and implement new preference optimization loss functions based on previously-evaluated performance metrics. This process leads to the discovery of previously-unknown and performant preference optimization algorithms. The best performing of these we call Discovered Preference Optimization (DiscoPOP), a novel algorithm that adaptively blends logistic and exponential losses. Experiments demonstrate the state-of-the-art performance of DiscoPOP and its successful transfer to held-out tasks. △ Less

Submitted 12 June, 2024; originally announced June 2024.

Multi-View People Detection in Large Scenes via Supervised View-Wise Contribution Weighting

Authors: Qi Zhang, Yunfei Gong, Daijie Chen, Antoni B. Chan, Hui Huang

Abstract: Recent deep learning-based multi-view people detection (MVD) methods have shown promising results on existing datasets. However, current methods are mainly trained and evaluated on small, single scenes with a limited number of multi-view frames and fixed camera views. As a result, these methods may not be practical for detecting people in larger, more complex scenes with severe occlusions and came… ▽ More Recent deep learning-based multi-view people detection (MVD) methods have shown promising results on existing datasets. However, current methods are mainly trained and evaluated on small, single scenes with a limited number of multi-view frames and fixed camera views. As a result, these methods may not be practical for detecting people in larger, more complex scenes with severe occlusions and camera calibration errors. This paper focuses on improving multi-view people detection by developing a supervised view-wise contribution weighting approach that better fuses multi-camera information under large scenes. Besides, a large synthetic dataset is adopted to enhance the model's generalization ability and enable more practical evaluation and comparison. The model's performance on new testing scenes is further improved with a simple domain adaptation technique. Experimental results demonstrate the effectiveness of our approach in achieving promising cross-scene multi-view people detection performance. See code here: https://vcc.tech/research/2024/MVD. △ Less

Submitted 30 May, 2024; originally announced May 2024.

Comments: AAAI 2024

The Pitfalls and Promise of Conformal Inference Under Adversarial Attacks

Authors: Ziquan Liu, Yufei Cui, Yan Yan, Yi Xu, Xiangyang Ji, Xue Liu, Antoni B. Chan

Abstract: In safety-critical applications such as medical imaging and autonomous driving, where decisions have profound implications for patient health and road safety, it is imperative to maintain both high adversarial robustness to protect against potential adversarial attacks and reliable uncertainty quantification in decision-making. With extensive research focused on enhancing adversarial robustness th… ▽ More In safety-critical applications such as medical imaging and autonomous driving, where decisions have profound implications for patient health and road safety, it is imperative to maintain both high adversarial robustness to protect against potential adversarial attacks and reliable uncertainty quantification in decision-making. With extensive research focused on enhancing adversarial robustness through various forms of adversarial training (AT), a notable knowledge gap remains concerning the uncertainty inherent in adversarially trained models. To address this gap, this study investigates the uncertainty of deep learning models by examining the performance of conformal prediction (CP) in the context of standard adversarial attacks within the adversarial defense community. It is first unveiled that existing CP methods do not produce informative prediction sets under the commonly used $l_{\infty}$-norm bounded attack if the model is not adversarially trained, which underpins the importance of adversarial training for CP. Our paper next demonstrates that the prediction set size (PSS) of CP using adversarially trained models with AT variants is often worse than using standard AT, inspiring us to research into CP-efficient AT for improved PSS. We propose to optimize a Beta-weighting loss with an entropy minimization regularizer during AT to improve CP-efficiency, where the Beta-weighting loss is shown to be an upper bound of PSS at the population level by our theoretical analysis. Moreover, our empirical study on four image classification datasets across three popular AT baselines validates the effectiveness of the proposed Uncertainty-Reducing AT (AT-UR). △ Less

Submitted 14 May, 2024; originally announced May 2024.

Comments: ICML2024

A Classification-Based Adaptive Segmentation Pipeline: Feasibility Study Using Polycystic Liver Disease and Metastases from Colorectal Cancer CT Images

Authors: Peilong Wang, Timothy L. Kline, Andy D. Missert, Cole J. Cook, Matthew R. Callstrom, Alex Chan, Robert P. Hartman, Zachary S. Kelm, Panagiotis Korfiatis

Abstract: Automated segmentation tools often encounter accuracy and adaptability issues when applied to images of different pathology. The purpose of this study is to explore the feasibility of building a workflow to efficiently route images to specifically trained segmentation models. By implementing a deep learning classifier to automatically classify the images and route them to appropriate segmentation… ▽ More Automated segmentation tools often encounter accuracy and adaptability issues when applied to images of different pathology. The purpose of this study is to explore the feasibility of building a workflow to efficiently route images to specifically trained segmentation models. By implementing a deep learning classifier to automatically classify the images and route them to appropriate segmentation models, we hope that our workflow can segment the images with different pathology accurately. The data we used in this study are 350 CT images from patients affected by polycystic liver disease and 350 CT images from patients presenting with liver metastases from colorectal cancer. All images had the liver manually segmented by trained imaging analysts. Our proposed adaptive segmentation workflow achieved a statistically significant improvement for the task of total liver segmentation compared to the generic single segmentation model (non-parametric Wilcoxon signed rank test, n=100, p-value << 0.001). This approach is applicable in a wide range of scenarios and should prove useful in clinical implementations of segmentation pipelines. △ Less

Submitted 2 May, 2024; originally announced May 2024.

Comments: J Digit Imaging. Inform. med. (2024)

Spectral form factor in chaotic, localized, and integrable open quantum many-body systems

Authors: Jiachen Li, Stephen Yan, Tomaž Prosen, Amos Chan

Abstract: We numerically study the spectral statistics of open quantum many-body systems (OQMBS) as signatures of quantum chaos (or the lack thereof), using the dissipative spectral form factor (DSFF), a generalization of the spectral form factor to complex spectra. We show that the DSFF of chaotic OQMBS generically displays the $\textit{quadratic}$ ramp-plateau behaviour of the Ginibre ensemble from random… ▽ More We numerically study the spectral statistics of open quantum many-body systems (OQMBS) as signatures of quantum chaos (or the lack thereof), using the dissipative spectral form factor (DSFF), a generalization of the spectral form factor to complex spectra. We show that the DSFF of chaotic OQMBS generically displays the $\textit{quadratic}$ ramp-plateau behaviour of the Ginibre ensemble from random matrix theory, in contrast to the linear ramp-plateau behaviour of the Gaussian ensemble in closed quantum systems. Furthermore, in the presence of many-body interactions, such RMT behaviour emerges only after a time scale $τ_{\mathrm{dev}}$, which generally increases with system size for sufficiently large system size, and can be identified as the non-Hermitian analogue of the $\textit{many-body Thouless time}$. The universality of the random matrix theory behavior is demonstrated by surveying twelve models of OQMBS, including random Kraus circuits (quantum channels) and random Lindbladians (Liouvillians) in several symmetry classes, as well as Lindbladians of paradigmatic models such as the Sachdev-Ye-Kitaev (SYK), XXZ, and the transverse field Ising models. We devise an unfolding and filtering procedure to remove variations of the averaged density of states which would otherwise hide the universal RMT-like signatures in the DSFF for chaotic OQMBS. Beyond chaotic OQMBS, we study the spectral statistics of non-chaotic OQMBS, specifically the integrable XX model and a system in the many-body localized (MBL) regime in the presence of dissipation, which exhibit DSFF behaviours distinct from the ramp-plateau behaviour of random matrix theory. Lastly, we study the DSFF of Lindbladians with the Hamiltonian term set to zero, i.e. only the jump operators are present, and demonstrate that the results of RMT universality and scaling of many-body Thouless time survive even without coherent evolution. △ Less

Submitted 2 May, 2024; originally announced May 2024.

A generalization of quantum pair state transfer

Authors: Sooyeong Kim, Hermie Monterde, Bahman Ahmadi, Ada Chan, Stephen Kirkland, Sarah Plosker

Abstract: An $s$-pair state in a graph is a quantum state of the form $\mathbf{e}_u+s\mathbf{e}_v$, where $u$ and $v$ are vertices in the graph and $s$ is a non-zero complex number. If $s=-1$ (resp., $s=1$), then such a state is called a pair state (resp. plus state). In this paper, we develop the theory of perfect $s$-pair state transfer in continuous quantum walks, where the Hamiltonian is taken to be the… ▽ More An $s$-pair state in a graph is a quantum state of the form $\mathbf{e}_u+s\mathbf{e}_v$, where $u$ and $v$ are vertices in the graph and $s$ is a non-zero complex number. If $s=-1$ (resp., $s=1$), then such a state is called a pair state (resp. plus state). In this paper, we develop the theory of perfect $s$-pair state transfer in continuous quantum walks, where the Hamiltonian is taken to be the adjacency, Laplacian or signless Laplacian matrix of the graph. We characterize perfect $s$-pair state transfer in complete graphs, cycles and antipodal distance-regular graphs admitting vertex perfect state transfer. We construct infinite families of graphs with perfect $s$-pair state transfer using quotient graphs and graphs that admit fractional revival. We provide necessary and sufficient conditions such that perfect state transfer between vertices in the line graph relative to the adjacency matrix is equivalent to perfect state transfer between the plus states formed by corresponding edges in the graph relative to the signless Laplacian matrix. Finally, we characterize perfect state transfer between vertices in the line graphs of Cartesian products relative to the adjacency matrix. △ Less

Submitted 28 July, 2024; v1 submitted 25 April, 2024; originally announced April 2024.

MSC Class: 05C50; 81P45; 05C76; 15A18; 81Q10

FreeDiff: Progressive Frequency Truncation for Image Editing with Diffusion Models

Authors: Wei Wu, Qingnan Fan, Shuai Qin, Hong Gu, Ruoyu Zhao, Antoni B. Chan

Abstract: Precise image editing with text-to-image models has attracted increasing interest due to their remarkable generative capabilities and user-friendly nature. However, such attempts face the pivotal challenge of misalignment between the intended precise editing target regions and the broader area impacted by the guidance in practice. Despite excellent methods leveraging attention mechanisms that have… ▽ More Precise image editing with text-to-image models has attracted increasing interest due to their remarkable generative capabilities and user-friendly nature. However, such attempts face the pivotal challenge of misalignment between the intended precise editing target regions and the broader area impacted by the guidance in practice. Despite excellent methods leveraging attention mechanisms that have been developed to refine the editing guidance, these approaches necessitate modifications through complex network architecture and are limited to specific editing tasks. In this work, we re-examine the diffusion process and misalignment problem from a frequency perspective, revealing that, due to the power law of natural images and the decaying noise schedule, the denoising network primarily recovers low-frequency image components during the earlier timesteps and thus brings excessive low-frequency signals for editing. Leveraging this insight, we introduce a novel fine-tuning free approach that employs progressive $\textbf{Fre}$qu$\textbf{e}$ncy truncation to refine the guidance of $\textbf{Diff}$usion models for universal editing tasks ($\textbf{FreeDiff}$). Our method achieves comparable results with state-of-the-art methods across a variety of editing tasks and on a diverse set of images, highlighting its potential as a versatile tool in image editing applications. △ Less

Submitted 18 April, 2024; originally announced April 2024.

Universal distributions of overlaps from unitary dynamics in generic quantum many-body systems

Authors: Alexios Christopoulos, Amos Chan, Andrea De Luca

Abstract: We study the preparation of a quantum state using a circuit of depth $t$ from a factorized state of $N$ sites. We argue that in the appropriate scaling limit of large $t$ and $N$, the overlap between states evolved under generic many-body chaotic dynamics belongs to a family of universal distribution that generalizes the celebrated Porter-Thomas distribution. This is a consequence of a mapping in… ▽ More We study the preparation of a quantum state using a circuit of depth $t$ from a factorized state of $N$ sites. We argue that in the appropriate scaling limit of large $t$ and $N$, the overlap between states evolved under generic many-body chaotic dynamics belongs to a family of universal distribution that generalizes the celebrated Porter-Thomas distribution. This is a consequence of a mapping in the space of replicas to a model of dilute domain walls. Our result provides a rare example in which analysis at an arbitrary number of replicas is possible, giving rise to the complete overlap distribution. Our general picture is derived and corroborated by the exact solution of the random phase model and of an emergent random matrix model given by the Ginibre ensemble. Finally, numerical simulations of two distinct random circuits show excellent agreement, thereby demonstrating universality. △ Less

Submitted 15 April, 2024; originally announced April 2024.

Comments: 15 pages, 6 figures

Foundational Challenges in Assuring Alignment and Safety of Large Language Models

Authors: Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L. Edelman, Zhaowei Zhang, Mario Günther, Anton Korinek, Jose Hernandez-Orallo, Lewis Hammond, Eric Bigelow, Alexander Pan, Lauro Langosco, Tomasz Korbak, Heidi Zhang, Ruiqi Zhong, Seán Ó hÉigeartaigh, Gabriel Recchia, Giulio Corsi , et al. (13 additional authors not shown)

Abstract: This work identifies 18 foundational challenges in assuring the alignment and safety of large language models (LLMs). These challenges are organized into three different categories: scientific understanding of LLMs, development and deployment methods, and sociotechnical challenges. Based on the identified challenges, we pose $200+$ concrete research questions. This work identifies 18 foundational challenges in assuring the alignment and safety of large language models (LLMs). These challenges are organized into three different categories: scientific understanding of LLMs, development and deployment methods, and sociotechnical challenges. Based on the identified challenges, we pose $200+$ concrete research questions. △ Less

Submitted 15 April, 2024; originally announced April 2024.

Learning Tracking Representations from Single Point Annotations

Authors: Qiangqiang Wu, Antoni B. Chan

Abstract: Existing deep trackers are typically trained with largescale video frames with annotated bounding boxes. However, these bounding boxes are expensive and time-consuming to annotate, in particular for large scale datasets. In this paper, we propose to learn tracking representations from single point annotations (i.e., 4.5x faster to annotate than the traditional bounding box) in a weakly supervised… ▽ More Existing deep trackers are typically trained with largescale video frames with annotated bounding boxes. However, these bounding boxes are expensive and time-consuming to annotate, in particular for large scale datasets. In this paper, we propose to learn tracking representations from single point annotations (i.e., 4.5x faster to annotate than the traditional bounding box) in a weakly supervised manner. Specifically, we propose a soft contrastive learning (SoCL) framework that incorporates target objectness prior into end-to-end contrastive learning. Our SoCL consists of adaptive positive and negative sample generation, which is memory-efficient and effective for learning tracking representations. We apply the learned representation of SoCL to visual tracking and show that our method can 1) achieve better performance than the fully supervised baseline trained with box annotations under the same annotation time cost; 2) achieve comparable performance of the fully supervised baseline by using the same number of training frames and meanwhile reducing annotation time cost by 78% and total fees by 85%; 3) be robust to annotation noise. △ Less

Submitted 15 April, 2024; originally announced April 2024.

Comments: Accept to CVPR2024-L3DIVU

Measuring Spectral Form Factor in Many-Body Chaotic and Localized Phases of Quantum Processors

Authors: Hang Dong, Pengfei Zhang, Ceren B. Dag, Yu Gao, Ning Wang, Jinfeng Deng, Xu Zhang, Jiachen Chen, Shibo Xu, Ke Wang, Yaozu Wu, Chuanyu Zhang, Feitong Jin, Xuhao Zhu, Aosai Zhang, Yiren Zou, Ziqi Tan, Zhengyi Cui, Zitian Zhu, Fanhao Shen, Tingting Li, Jiarun Zhong, Zehang Bao, Hekang Li, Zhen Wang , et al. (6 additional authors not shown)

Abstract: The spectral form factor (SFF) captures universal spectral fluctuations as signatures of quantum chaos, and has been instrumental in advancing multiple frontiers of physics including the studies of black holes and quantum many-body systems. However, the measurement of SFF in many-body systems is challenging due to the difficulty in resolving level spacings that become exponentially small with incr… ▽ More The spectral form factor (SFF) captures universal spectral fluctuations as signatures of quantum chaos, and has been instrumental in advancing multiple frontiers of physics including the studies of black holes and quantum many-body systems. However, the measurement of SFF in many-body systems is challenging due to the difficulty in resolving level spacings that become exponentially small with increasing system size. Here we experimentally measure the SFF to probe the presence or absence of chaos in quantum many-body systems using a superconducting quantum processor with a randomized measurement protocol. For a Floquet chaotic system, we observe signatures of spectral rigidity of random matrix theory in SFF given by the ramp-plateau behavior. For a Hamiltonian system, we utilize SFF to distinguish the quantum many-body chaotic phase and the prethermal many-body localization. We observe the dip-ramp-plateau behavior of random matrix theory in the chaotic phase, and contrast the scaling of the plateau time in system size between the many-body chaotic and localized phases. Furthermore, we probe the eigenstate statistics by measuring a generalization of the SFF, known as the partial SFF, and observe distinct behaviors in the purities of the reduced density matrix in the two phases. This work unveils a new way of extracting the universal signatures of many-body quantum chaos in quantum devices by probing the correlations in eigenenergies and eigenstates. △ Less

Submitted 25 March, 2024; originally announced March 2024.

Comments: 12 pages, 9 figures

Anytime, Anywhere, Anyone: Investigating the Feasibility of Segment Anything Model for Crowd-Sourcing Medical Image Annotations

Authors: Pranav Kulkarni, Adway Kanhere, Dharmam Savani, Andrew Chan, Devina Chatterjee, Paul H. Yi, Vishwa S. Parekh

Abstract: Curating annotations for medical image segmentation is a labor-intensive and time-consuming task that requires domain expertise, resulting in "narrowly" focused deep learning (DL) models with limited translational utility. Recently, foundation models like the Segment Anything Model (SAM) have revolutionized semantic segmentation with exceptional zero-shot generalizability across various domains, i… ▽ More Curating annotations for medical image segmentation is a labor-intensive and time-consuming task that requires domain expertise, resulting in "narrowly" focused deep learning (DL) models with limited translational utility. Recently, foundation models like the Segment Anything Model (SAM) have revolutionized semantic segmentation with exceptional zero-shot generalizability across various domains, including medical imaging, and hold a lot of promise for streamlining the annotation process. However, SAM has yet to be evaluated in a crowd-sourced setting to curate annotations for training 3D DL segmentation models. In this work, we explore the potential of SAM for crowd-sourcing "sparse" annotations from non-experts to generate "dense" segmentation masks for training 3D nnU-Net models, a state-of-the-art DL segmentation model. Our results indicate that while SAM-generated annotations exhibit high mean Dice scores compared to ground-truth annotations, nnU-Net models trained on SAM-generated annotations perform significantly worse than nnU-Net models trained on ground-truth annotations ($p<0.001$, all). △ Less

Submitted 22 March, 2024; originally announced March 2024.

GPT-4V(ision) Unsuitable for Clinical Care and Education: A Clinician-Evaluated Assessment

Authors: Senthujan Senkaiahliyan, Augustin Toma, Jun Ma, An-Wen Chan, Andrew Ha, Kevin R. An, Hrishikesh Suresh, Barry Rubin, Bo Wang

Abstract: OpenAI's large multimodal model, GPT-4V(ision), was recently developed for general image interpretation. However, less is known about its capabilities with medical image interpretation and diagnosis. Board-certified physicians and senior residents assessed GPT-4V's proficiency across a range of medical conditions using imaging modalities such as CT scans, MRIs, ECGs, and clinical photographs. Alth… ▽ More OpenAI's large multimodal model, GPT-4V(ision), was recently developed for general image interpretation. However, less is known about its capabilities with medical image interpretation and diagnosis. Board-certified physicians and senior residents assessed GPT-4V's proficiency across a range of medical conditions using imaging modalities such as CT scans, MRIs, ECGs, and clinical photographs. Although GPT-4V is able to identify and explain medical images, its diagnostic accuracy and clinical decision-making abilities are poor, posing risks to patient safety. Despite the potential that large language models may have in enhancing medical education and delivery, the current limitations of GPT-4V in interpreting medical images reinforces the importance of appropriate caution when using it for clinical decision-making. △ Less

Submitted 14 November, 2023; originally announced March 2024.

A Fixed-Point Approach to Unified Prompt-Based Counting

Authors: Wei Lin, Antoni B. Chan

Abstract: Existing class-agnostic counting models typically rely on a single type of prompt, e.g., box annotations. This paper aims to establish a comprehensive prompt-based counting framework capable of generating density maps for concerned objects indicated by various prompt types, such as box, point, and text. To achieve this goal, we begin by converting prompts from different modalities into prompt mask… ▽ More Existing class-agnostic counting models typically rely on a single type of prompt, e.g., box annotations. This paper aims to establish a comprehensive prompt-based counting framework capable of generating density maps for concerned objects indicated by various prompt types, such as box, point, and text. To achieve this goal, we begin by converting prompts from different modalities into prompt masks without requiring training. These masks are then integrated into a class-agnostic counting methodology for predicting density maps. Furthermore, we introduce a fixed-point inference along with an associated loss function to improve counting accuracy, all without introducing new parameters. The effectiveness of this method is substantiated both theoretically and experimentally. Additionally, a contrastive training scheme is implemented to mitigate dataset bias inherent in current class-agnostic counting datasets, a strategy whose effectiveness is confirmed by our ablation study. Our model excels in prominent class-agnostic datasets and exhibits superior performance in cross-dataset adaptation tasks. △ Less

Submitted 15 March, 2024; originally announced March 2024.

Comments: Accepted by AAAI 2024

Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation

Authors: Marcel Torne, Anthony Simeonov, Zechu Li, April Chan, Tao Chen, Abhishek Gupta, Pulkit Agrawal

Abstract: Imitation learning methods need significant human supervision to learn policies robust to changes in object poses, physical disturbances, and visual distractors. Reinforcement learning, on the other hand, can explore the environment autonomously to learn robust behaviors but may require impractical amounts of unsafe real-world data collection. To learn performant, robust policies without the burde… ▽ More Imitation learning methods need significant human supervision to learn policies robust to changes in object poses, physical disturbances, and visual distractors. Reinforcement learning, on the other hand, can explore the environment autonomously to learn robust behaviors but may require impractical amounts of unsafe real-world data collection. To learn performant, robust policies without the burden of unsafe real-world data collection or extensive human supervision, we propose RialTo, a system for robustifying real-world imitation learning policies via reinforcement learning in "digital twin" simulation environments constructed on the fly from small amounts of real-world data. To enable this real-to-sim-to-real pipeline, RialTo proposes an easy-to-use interface for quickly scanning and constructing digital twins of real-world environments. We also introduce a novel "inverse distillation" procedure for bringing real-world demonstrations into simulated environments for efficient fine-tuning, with minimal human intervention and engineering required. We evaluate RialTo across a variety of robotic manipulation problems in the real world, such as robustly stacking dishes on a rack, placing books on a shelf, and six other tasks. RialTo increases (over 67%) in policy robustness without requiring extensive human data collection. Project website and videos at https://real-to-sim-to-real.github.io/RialTo/ △ Less

Submitted 6 March, 2024; originally announced March 2024.

Comments: Project page: https://real-to-sim-to-real.github.io/RialTo/

Robust Unsupervised Crowd Counting and Localization with Adaptive Resolution SAM

Authors: Jia Wan, Qiangqiang Wu, Wei Lin, Antoni B. Chan

Abstract: The existing crowd counting models require extensive training data, which is time-consuming to annotate. To tackle this issue, we propose a simple yet effective crowd counting method by utilizing the Segment-Everything-Everywhere Model (SEEM), an adaptation of the Segmentation Anything Model (SAM), to generate pseudo-labels for training crowd counting models. However, our initial investigation rev… ▽ More The existing crowd counting models require extensive training data, which is time-consuming to annotate. To tackle this issue, we propose a simple yet effective crowd counting method by utilizing the Segment-Everything-Everywhere Model (SEEM), an adaptation of the Segmentation Anything Model (SAM), to generate pseudo-labels for training crowd counting models. However, our initial investigation reveals that SEEM's performance in dense crowd scenes is limited, primarily due to the omission of many persons in high-density areas. To overcome this limitation, we propose an adaptive resolution SEEM to handle the scale variations, occlusions, and overlapping of people within crowd scenes. Alongside this, we introduce a robust localization method, based on Gaussian Mixture Models, for predicting the head positions in the predicted people masks. Given the mask and point pseudo-labels, we propose a robust loss function, which is designed to exclude uncertain regions based on SEEM's predictions, thereby enhancing the training process of the counting networks. Finally, we propose an iterative method for generating pseudo-labels. This method aims at improving the quality of the segmentation masks by identifying more tiny persons in high-density regions, which are often missed in the first pseudo-labeling stage. Overall, our proposed method achieves the best unsupervised performance in crowd counting, while also being comparable results to some supervised methods. This makes it a highly effective and versatile tool for crowd counting, especially in situations where labeled data is not available. △ Less

Submitted 27 February, 2024; originally announced February 2024.

Projected state ensemble of a generic model of many-body quantum chaos

Authors: Amos Chan, Andrea De Luca

Abstract: The projected ensemble is based on the study of the quantum state of a subsystem $A$ conditioned on projective measurements in its complement. Recent studies have observed that a more refined measure of the thermalization of a chaotic quantum system can be defined on the basis of convergence of the projected ensemble to a quantum state design, i.e. a system thermalizes when it becomes indistinguis… ▽ More The projected ensemble is based on the study of the quantum state of a subsystem $A$ conditioned on projective measurements in its complement. Recent studies have observed that a more refined measure of the thermalization of a chaotic quantum system can be defined on the basis of convergence of the projected ensemble to a quantum state design, i.e. a system thermalizes when it becomes indistinguishable, up to the $k$-th moment, from a Haar ensemble of uniformly distributed pure states. Here we consider a random unitary circuit with the brick-wall geometry and analyze its convergence to the Haar ensemble through the frame potential and its mapping to a statistical mechanical problem. This approach allows us to highlight a geometric interpretation of the frame potential based on the existence of a fluctuating membrane, similar to those appearing in the study of entanglement entropies. At large local Hilbert space dimension $q$, we find that all moments converge simultaneously with a time scaling linearly in the size of region $A$, a feature previously observed in dual unitary models. However, based on the geometric interpretation, we argue that the scaling at finite $q$ on the basis of rare membrane fluctuations, finding the logarithmic scaling of design times $t_k = O(\log k)$. Our results are supported with numerical simulations performed at $q=2$. △ Less

Submitted 26 February, 2024; originally announced February 2024.

Comments: 23 pages, 7 figures. Submitted for the special issue of Journal of Physics A: Mathematical and Theoretical on Quantum-Circuit Models for Many-Body Physics Out of Equilibrium

Copilot Evaluation Harness: Evaluating LLM-Guided Software Programming

Authors: Anisha Agarwal, Aaron Chan, Shubham Chandel, Jinu Jang, Shaun Miller, Roshanak Zilouchian Moghaddam, Yevhen Mohylevskyy, Neel Sundaresan, Michele Tufano

Abstract: The integration of Large Language Models (LLMs) into Development Environments (IDEs) has become a focal point in modern software development. LLMs such as OpenAI GPT-3.5/4 and Code Llama offer the potential to significantly augment developer productivity by serving as intelligent, chat-driven programming assistants. However, utilizing LLMs out of the box is unlikely to be optimal for any given sce… ▽ More The integration of Large Language Models (LLMs) into Development Environments (IDEs) has become a focal point in modern software development. LLMs such as OpenAI GPT-3.5/4 and Code Llama offer the potential to significantly augment developer productivity by serving as intelligent, chat-driven programming assistants. However, utilizing LLMs out of the box is unlikely to be optimal for any given scenario. Rather, each system requires the LLM to be honed to its set of heuristics to ensure the best performance. In this paper, we introduce the Copilot evaluation harness: a set of data and tools for evaluating LLM-guided IDE interactions, covering various programming scenarios and languages. We propose our metrics as a more robust and information-dense evaluation than previous state of the art evaluation systems. We design and compute both static and execution based success metrics for scenarios encompassing a wide range of developer tasks, including code generation from natural language (generate), documentation generation from code (doc), test case generation (test), bug-fixing (fix), and workspace understanding and query resolution (workspace). These success metrics are designed to evaluate the performance of LLMs within a given IDE and its respective parameter space. Our learnings from evaluating three common LLMs using these metrics can inform the development and validation of future scenarios in LLM guided IDEs. △ Less

Submitted 21 February, 2024; originally announced February 2024.

Designing interactive data visualizations representing recovery progress for patients after stroke

Authors: Alicia Ouskine, Adrian D. C. Chan, Fateme Rajabiyazdi

Abstract: Stroke is one of the leading causes of disability worldwide. The efficacy of recovery is determined by a variety of factors, including patient adherence to rehabilitation programs. One way to increase patient adherence to their rehabilitation program is to show patients their progress that is visualized in a simple and intuitive way. We begin to gather preliminary information on Functional Capacit… ▽ More Stroke is one of the leading causes of disability worldwide. The efficacy of recovery is determined by a variety of factors, including patient adherence to rehabilitation programs. One way to increase patient adherence to their rehabilitation program is to show patients their progress that is visualized in a simple and intuitive way. We begin to gather preliminary information on Functional Capacity, Motor Function, and Mood/cognition from occupational Therapists at the Bruyere Hospital to gain a better understanding of how stroke recovery data is collected within in-patient stroke rehabilitation centers. The future aim is to design, develop, and evaluate a data visualization tool representing progress made by patients recovering from stroke. △ Less

Submitted 18 February, 2024; originally announced February 2024.

Comments: 2 pages

Hidden in Plain Sight: Undetectable Adversarial Bias Attacks on Vulnerable Patient Populations

Authors: Pranav Kulkarni, Andrew Chan, Nithya Navarathna, Skylar Chan, Paul H. Yi, Vishwa S. Parekh

Abstract: The proliferation of artificial intelligence (AI) in radiology has shed light on the risk of deep learning (DL) models exacerbating clinical biases towards vulnerable patient populations. While prior literature has focused on quantifying biases exhibited by trained DL models, demographically targeted adversarial bias attacks on DL models and its implication in the clinical environment remains an u… ▽ More The proliferation of artificial intelligence (AI) in radiology has shed light on the risk of deep learning (DL) models exacerbating clinical biases towards vulnerable patient populations. While prior literature has focused on quantifying biases exhibited by trained DL models, demographically targeted adversarial bias attacks on DL models and its implication in the clinical environment remains an underexplored field of research in medical imaging. In this work, we demonstrate that demographically targeted label poisoning attacks can introduce undetectable underdiagnosis bias in DL models. Our results across multiple performance metrics and demographic groups like sex, age, and their intersectional subgroups show that adversarial bias attacks demonstrate high-selectivity for bias in the targeted group by degrading group model performance without impacting overall model performance. Furthermore, our results indicate that adversarial bias attacks result in biased DL models that propagate prediction bias even when evaluated with external datasets. △ Less

Submitted 7 April, 2024; v1 submitted 8 February, 2024; originally announced February 2024.

Comments: 29 pages, 4 figures

ReviewFlow: Intelligent Scaffolding to Support Academic Peer Reviewing

Authors: Lu Sun, Aaron Chan, Yun Seo Chang, Steven P. Dow

Abstract: Peer review is a cornerstone of science. Research communities conduct peer reviews to assess contributions and to improve the overall quality of science work. Every year, new community members are recruited as peer reviewers for the first time. How could technology help novices adhere to their community's practices and standards for peer reviewing? To better understand peer review practices and ch… ▽ More Peer review is a cornerstone of science. Research communities conduct peer reviews to assess contributions and to improve the overall quality of science work. Every year, new community members are recruited as peer reviewers for the first time. How could technology help novices adhere to their community's practices and standards for peer reviewing? To better understand peer review practices and challenges, we conducted a formative study with 10 novices and 10 experts. We found that many experts adopt a workflow of annotating, note-taking, and synthesizing notes into well-justified reviews that align with community standards. Novices lack timely guidance on how to read and assess submissions and how to structure paper reviews. To support the peer review process, we developed ReviewFlow -- an AI-driven workflow that scaffolds novices with contextual reflections to critique and annotate submissions, in-situ knowledge support to assess novelty, and notes-to-outline synthesis to help align peer reviews with community expectations. In a within-subjects experiment, 16 inexperienced reviewers wrote reviews in two conditions: using ReviewFlow and using a baseline environment with minimal guidance. With ReviewFlow, participants produced more comprehensive reviews, identifying more pros and cons. While participants appreciated the streamlined process support from ReviewFlow, they also expressed concerns about using AI as part of the scientific review process. We discuss the implications of using AI to scaffold the peer review process on scientific work and beyond. △ Less

Submitted 26 February, 2024; v1 submitted 5 February, 2024; originally announced February 2024.

Comments: 19 pages, accepted at the 29th ACM Conference on Intelligent User Interfaces (IUI 2024)

Hyper-Diffusion: Estimating Epistemic and Aleatoric Uncertainty with a Single Model

Authors: Matthew A. Chan, Maria J. Molina, Christopher A. Metzler

Abstract: Estimating and disentangling epistemic uncertainty (uncertainty that can be reduced with more training data) and aleatoric uncertainty (uncertainty that is inherent to the task at hand) is critically important when applying machine learning (ML) to high-stakes applications such as medical imaging and weather forecasting. Conditional diffusion models' breakthrough ability to accurately and efficien… ▽ More Estimating and disentangling epistemic uncertainty (uncertainty that can be reduced with more training data) and aleatoric uncertainty (uncertainty that is inherent to the task at hand) is critically important when applying machine learning (ML) to high-stakes applications such as medical imaging and weather forecasting. Conditional diffusion models' breakthrough ability to accurately and efficiently sample from the posterior distribution of a dataset now makes uncertainty estimation conceptually straightforward: One need only train and sample from a large ensemble of diffusion models. Unfortunately, training such an ensemble becomes computationally intractable as the complexity of the model architecture grows. In this work we introduce a new approach to ensembling, hyper-diffusion, which allows one to accurately estimate epistemic and aleatoric uncertainty with a single model. Unlike existing Monte Carlo dropout based single-model ensembling methods, hyper-diffusion offers the same prediction accuracy as multi-model ensembles. We validate our approach on two distinct tasks: x-ray computed tomography (CT) reconstruction and weather temperature forecasting. △ Less

Submitted 5 February, 2024; originally announced February 2024.

Comments: 10 pages, 7 figures

Dense Reward for Free in Reinforcement Learning from Human Feedback

Authors: Alex J. Chan, Hao Sun, Samuel Holt, Mihaela van der Schaar

Abstract: Reinforcement Learning from Human Feedback (RLHF) has been credited as the key advance that has allowed Large Language Models (LLMs) to effectively follow instructions and produce useful assistance. Classically, this involves generating completions from the LLM in response to a query before using a separate reward model to assign a score to the full completion. As an auto-regressive process, the L… ▽ More Reinforcement Learning from Human Feedback (RLHF) has been credited as the key advance that has allowed Large Language Models (LLMs) to effectively follow instructions and produce useful assistance. Classically, this involves generating completions from the LLM in response to a query before using a separate reward model to assign a score to the full completion. As an auto-regressive process, the LLM has to take many "actions" (selecting individual tokens) and only receives a single, sparse reward at the end of an episode, a setup that is known to be difficult to optimise in traditional reinforcement learning. In this work we leverage the fact that the reward model contains more information than just its scalar output, in particular, it calculates an attention map over tokens as part of the transformer architecture. We use these attention weights to redistribute the reward along the whole completion, effectively densifying the signal and highlighting the most important tokens, all without incurring extra computational cost or requiring any additional modelling. We demonstrate that, theoretically, this approach is equivalent to potential-based reward shaping, ensuring that the optimal policy remains unchanged. Empirically, we show that it stabilises training, accelerates the rate of learning, and, in practical cases, may lead to better local optima. △ Less

Submitted 1 February, 2024; originally announced February 2024.

Dirac mass induced by optical gain and loss

Authors: Letian Yu, Haoran Xue, Ruixiang Guo, Eng Aik Chan, Yun Yong Terh, Cesare Soci, Baile Zhang, Y. D. Chong

Abstract: Mass is commonly regarded as an intrinsic property of matter, but modern physics reveals particle masses to have complex origins, such as the Higgs mechanism in high-energy physics. In crystal lattices such as graphene, relativistic Dirac particles can exist as low-energy quasiparticles with masses imparted by lattice symmetry-breaking perturbations. These mass-generating mechanisms all assume Her… ▽ More Mass is commonly regarded as an intrinsic property of matter, but modern physics reveals particle masses to have complex origins, such as the Higgs mechanism in high-energy physics. In crystal lattices such as graphene, relativistic Dirac particles can exist as low-energy quasiparticles with masses imparted by lattice symmetry-breaking perturbations. These mass-generating mechanisms all assume Hermiticity, or the conservation of energy in detail. Using a photonic synthetic lattice, we show experimentally that Dirac masses can be generated via non-Hermitian perturbations based on optical gain and loss. We then explore how the space-time engineering of the gain/loss-induced Dirac mass affects the quasiparticles. As we show, the quasiparticles undergo Klein tunnelling at spatial boundaries, but a local breaking of a non-Hermitian symmetry can produce a novel flux nonconservation effect at the domain walls. At a temporal boundary that abruptly flips the sign of the Dirac mass, we observe a variant of the time reflection phenomenon: in the nonrelativistic limit, the Dirac quasiparticle reverses its velocity, while in the relativistic limit the original velocity is retained. △ Less

Submitted 14 April, 2024; v1 submitted 27 January, 2024; originally announced January 2024.

Black-Box Access is Insufficient for Rigorous AI Audits

Authors: Stephen Casper, Carson Ezell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Benjamin Bucknall, Andreas Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, Lee Sharkey, Satyapriya Krishna, Marvin Von Hagen, Silas Alberti, Alan Chan, Qinyi Sun, Michael Gerovitch, David Bau, Max Tegmark, David Krueger, Dylan Hadfield-Menell

Abstract: External audits of AI systems are increasingly recognized as a key mechanism for AI governance. The effectiveness of an audit, however, depends on the degree of access granted to auditors. Recent audits of state-of-the-art AI systems have primarily relied on black-box access, in which auditors can only query the system and observe its outputs. However, white-box access to the system's inner workin… ▽ More External audits of AI systems are increasingly recognized as a key mechanism for AI governance. The effectiveness of an audit, however, depends on the degree of access granted to auditors. Recent audits of state-of-the-art AI systems have primarily relied on black-box access, in which auditors can only query the system and observe its outputs. However, white-box access to the system's inner workings (e.g., weights, activations, gradients) allows an auditor to perform stronger attacks, more thoroughly interpret models, and conduct fine-tuning. Meanwhile, outside-the-box access to training and deployment information (e.g., methodology, code, documentation, data, deployment details, findings from internal evaluations) allows auditors to scrutinize the development process and design more targeted evaluations. In this paper, we examine the limitations of black-box audits and the advantages of white- and outside-the-box audits. We also discuss technical, physical, and legal safeguards for performing these audits with minimal security risks. Given that different forms of access can lead to very different levels of evaluation, we conclude that (1) transparency regarding the access and methods used by auditors is necessary to properly interpret audit results, and (2) white- and outside-the-box access allow for substantially more scrutiny than black-box access alone. △ Less

Submitted 29 May, 2024; v1 submitted 25 January, 2024; originally announced January 2024.

Comments: FAccT 2024

Journal ref: The 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT '24), June 3-6, 2024, Rio de Janeiro, Brazil

Visibility into AI Agents

Authors: Alan Chan, Carson Ezell, Max Kaufmann, Kevin Wei, Lewis Hammond, Herbie Bradley, Emma Bluemke, Nitarshan Rajkumar, David Krueger, Noam Kolt, Lennart Heim, Markus Anderljung

Abstract: Increased delegation of commercial, scientific, governmental, and personal activities to AI agents -- systems capable of pursuing complex goals with limited supervision -- may exacerbate existing societal risks and introduce new risks. Understanding and mitigating these risks involves critically evaluating existing governance structures, revising and adapting these structures where needed, and ens… ▽ More Increased delegation of commercial, scientific, governmental, and personal activities to AI agents -- systems capable of pursuing complex goals with limited supervision -- may exacerbate existing societal risks and introduce new risks. Understanding and mitigating these risks involves critically evaluating existing governance structures, revising and adapting these structures where needed, and ensuring accountability of key stakeholders. Information about where, why, how, and by whom certain AI agents are used, which we refer to as visibility, is critical to these objectives. In this paper, we assess three categories of measures to increase visibility into AI agents: agent identifiers, real-time monitoring, and activity logging. For each, we outline potential implementations that vary in intrusiveness and informativeness. We analyze how the measures apply across a spectrum of centralized through decentralized deployment contexts, accounting for various actors in the supply chain including hardware and software service providers. Finally, we discuss the implications of our measures for privacy and concentration of power. Further work into understanding the measures and mitigating their negative impacts can help to build a foundation for the governance of AI agents. △ Less

Submitted 17 May, 2024; v1 submitted 23 January, 2024; originally announced January 2024.

Comments: Accepted to ACM Conference on Fairness, Accountability, and Transparency (ACM FAccT 2024)

Error Propagation Analysis for Multithreaded Programs: An Empirical Approach

Authors: Stefan Winter, Abraham Chan, Habib Saissi, Karthik Pattabiraman, Neeraj Suri

Abstract: Fault injection is a technique to measure the robustness of a program to errors by introducing faults into the program under test. Following a fault injection experiment, Error Propagation Analysis (EPA) is deployed to understand how errors affect a program's execution. EPA typically compares the traces of a fault-free (golden) run with those from a faulty run of the program. While this suffices f… ▽ More Fault injection is a technique to measure the robustness of a program to errors by introducing faults into the program under test. Following a fault injection experiment, Error Propagation Analysis (EPA) is deployed to understand how errors affect a program's execution. EPA typically compares the traces of a fault-free (golden) run with those from a faulty run of the program. While this suffices for deterministic programs, EPA approaches are unsound for multithreaded programs with non-deterministic golden runs. In this paper, we propose Invariant Propagation Analysis (IPA) as the use of automatically inferred likely invariants ("invariants" in the following) in lieu of golden traces for conducting EPA in multithreaded programs. We evaluate the stability and fault coverage of invariants derived by IPA through fault injection experiments across six different fault types and six representative programs that can be executed with varying numbers of threads. We find that stable invariants can be inferred in all cases, but their fault coverage depends on the application and the fault type. We also find that fault coverage for multithreaded executions with IPA can be even higher than for traditional singlethreaded EPA, which emphasizes that IPA results cannot be trivially extrapolated from traditional EPA results. △ Less

Submitted 27 December, 2023; originally announced December 2023.

Comments: Extended version of conference paper, originally published in the proceedings of ICST'17 (see: https://ieeexplore.ieee.org/document/7927974)

Hazards from Increasingly Accessible Fine-Tuning of Downloadable Foundation Models

Authors: Alan Chan, Ben Bucknall, Herbie Bradley, David Krueger

Abstract: Public release of the weights of pretrained foundation models, otherwise known as downloadable access \citep{solaiman_gradient_2023}, enables fine-tuning without the prohibitive expense of pretraining. Our work argues that increasingly accessible fine-tuning of downloadable models may increase hazards. First, we highlight research to improve the accessibility of fine-tuning. We split our discussio… ▽ More Public release of the weights of pretrained foundation models, otherwise known as downloadable access \citep{solaiman_gradient_2023}, enables fine-tuning without the prohibitive expense of pretraining. Our work argues that increasingly accessible fine-tuning of downloadable models may increase hazards. First, we highlight research to improve the accessibility of fine-tuning. We split our discussion into research that A) reduces the computational cost of fine-tuning and B) improves the ability to share that cost across more actors. Second, we argue that increasingly accessible fine-tuning methods may increase hazard through facilitating malicious use and making oversight of models with potentially dangerous capabilities more difficult. Third, we discuss potential mitigatory measures, as well as benefits of more accessible fine-tuning. Given substantial remaining uncertainty about hazards, we conclude by emphasizing the urgent need for the development of mitigations. △ Less

Submitted 22 December, 2023; originally announced December 2023.

Comments: Accepted as a spotlight workshop paper at the Socially Responsible Language Modelling Research (SoLaR) workshop, held at NeurIPS 2023

Harmonizing Global Voices: Culturally-Aware Models for Enhanced Content Moderation

Authors: Alex J. Chan, José Luis Redondo García, Fabrizio Silvestri, Colm O'Donnel, Konstantina Palla

Abstract: Content moderation at scale faces the challenge of considering local cultural distinctions when assessing content. While global policies aim to maintain decision-making consistency and prevent arbitrary rule enforcement, they often overlook regional variations in interpreting natural language as expressed in content. In this study, we are looking into how moderation systems can tackle this issue b… ▽ More Content moderation at scale faces the challenge of considering local cultural distinctions when assessing content. While global policies aim to maintain decision-making consistency and prevent arbitrary rule enforcement, they often overlook regional variations in interpreting natural language as expressed in content. In this study, we are looking into how moderation systems can tackle this issue by adapting to local comprehension nuances. We train large language models on extensive datasets of media news and articles to create culturally attuned models. The latter aim to capture the nuances of communication across geographies with the goal of recognizing cultural and societal variations in what is considered offensive content. We further explore the capability of these models to generate explanations for instances of content violation, aiming to shed light on how policy guidelines are perceived when cultural and societal contexts change. We find that training on extensive media datasets successfully induced cultural awareness and resulted in improvements in handling content violations on a regional basis. Additionally, these advancements include the ability to provide explanations that align with the specific local norms and nuances as evidenced by the annotators' preference in our conducted study. This multifaceted success reinforces the critical role of an adaptable content moderation approach in keeping pace with the ever-evolving nature of the content it oversees. △ Less

Submitted 4 December, 2023; originally announced December 2023.

Comments: 12 pages, 8 Figures. Supplementary material

When is Off-Policy Evaluation Useful? A Data-Centric Perspective

Authors: Hao Sun, Alex J. Chan, Nabeel Seedat, Alihan Hüyük, Mihaela van der Schaar

Abstract: Evaluating the value of a hypothetical target policy with only a logged dataset is important but challenging. On the one hand, it brings opportunities for safe policy improvement under high-stakes scenarios like clinical guidelines. On the other hand, such opportunities raise a need for precise off-policy evaluation (OPE). While previous work on OPE focused on improving the algorithm in value esti… ▽ More Evaluating the value of a hypothetical target policy with only a logged dataset is important but challenging. On the one hand, it brings opportunities for safe policy improvement under high-stakes scenarios like clinical guidelines. On the other hand, such opportunities raise a need for precise off-policy evaluation (OPE). While previous work on OPE focused on improving the algorithm in value estimation, in this work, we emphasize the importance of the offline dataset, hence putting forward a data-centric framework for evaluating OPE problems. We propose DataCOPE, a data-centric framework for evaluating OPE, that answers the questions of whether and to what extent we can evaluate a target policy given a dataset. DataCOPE (1) forecasts the overall performance of OPE algorithms without access to the environment, which is especially useful before real-world deployment where evaluating OPE is impossible; (2) identifies the sub-group in the dataset where OPE can be inaccurate; (3) permits evaluations of datasets or data-collection strategies for OPE problems. Our empirical analysis of DataCOPE in the logged contextual bandit settings using healthcare datasets confirms its ability to evaluate both machine-learning and human expert policies like clinical guidelines. △ Less

Submitted 23 November, 2023; originally announced November 2023.

Comments: Off-Policy Evaluation, Data-Centric AI, Data-Centric Reinforcement Learning, Reinforcement Learning

Retrieving positions of closely packed sub-wavelength nanoparticles from their diffraction patterns

Authors: Benquan Wang, Ruyi An, Eng Aik Chan, Giorgio Adamo, Jin-Kyu So, Yewen Li, Zexiang Shen, Bo An, Nikolay I. Zheludev

Abstract: Distinguishing two objects or point sources located closer than the Rayleigh distance is impossible in conventional microscopy. Understandably, the task becomes increasingly harder with a growing number of particles placed in close proximity. It has been recently demonstrated that subwavelength nanoparticles in closely packed clusters can be counted by AI-enabled analysis of the diffraction patter… ▽ More Distinguishing two objects or point sources located closer than the Rayleigh distance is impossible in conventional microscopy. Understandably, the task becomes increasingly harder with a growing number of particles placed in close proximity. It has been recently demonstrated that subwavelength nanoparticles in closely packed clusters can be counted by AI-enabled analysis of the diffraction patterns of coherent light scattered by the cluster. Here we show that deep learning analysis can determine the actual position of the nanoparticle in the cluster of subwavelength particles from a sing-shot diffraction pattern even if they are separated by distances below the Rayleigh resolution limit of a conventional microscope. △ Less

Submitted 17 November, 2023; originally announced November 2023.

Comments: 6 pages, 3 figures

Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectives

Authors: Elizabeth Seger, Noemi Dreksler, Richard Moulange, Emily Dardaman, Jonas Schuett, K. Wei, Christoph Winter, Mackenzie Arnold, Seán Ó hÉigeartaigh, Anton Korinek, Markus Anderljung, Ben Bucknall, Alan Chan, Eoghan Stafford, Leonie Koessler, Aviv Ovadya, Ben Garfinkel, Emma Bluemke, Michael Aird, Patrick Levermore, Julian Hazell, Abhishek Gupta

Abstract: Recent decisions by leading AI labs to either open-source their models or to restrict access to their models has sparked debate about whether, and how, increasingly capable AI models should be shared. Open-sourcing in AI typically refers to making model architecture and weights freely and publicly accessible for anyone to modify, study, build on, and use. This offers advantages such as enabling ex… ▽ More Recent decisions by leading AI labs to either open-source their models or to restrict access to their models has sparked debate about whether, and how, increasingly capable AI models should be shared. Open-sourcing in AI typically refers to making model architecture and weights freely and publicly accessible for anyone to modify, study, build on, and use. This offers advantages such as enabling external oversight, accelerating progress, and decentralizing control over AI development and use. However, it also presents a growing potential for misuse and unintended consequences. This paper offers an examination of the risks and benefits of open-sourcing highly capable foundation models. While open-sourcing has historically provided substantial net benefits for most software and AI development processes, we argue that for some highly capable foundation models likely to be developed in the near future, open-sourcing may pose sufficiently extreme risks to outweigh the benefits. In such a case, highly capable foundation models should not be open-sourced, at least not initially. Alternative strategies, including non-open-source model sharing options, are explored. The paper concludes with recommendations for developers, standard-setting bodies, and governments for establishing safe and responsible model sharing practices and preserving open-source benefits where safe. △ Less

Submitted 29 September, 2023; originally announced November 2023.

Comments: Official release at https://www.governance.ai/research-paper/open-sourcing-highly-capable-foundation-models

Optimising Human-AI Collaboration by Learning Convincing Explanations

Authors: Alex J. Chan, Alihan Huyuk, Mihaela van der Schaar

Abstract: Machine learning models are being increasingly deployed to take, or assist in taking, complicated and high-impact decisions, from quasi-autonomous vehicles to clinical decision support systems. This poses challenges, particularly when models have hard-to-detect failure modes and are able to take actions without oversight. In order to handle this challenge, we propose a method for a collaborative s… ▽ More Machine learning models are being increasingly deployed to take, or assist in taking, complicated and high-impact decisions, from quasi-autonomous vehicles to clinical decision support systems. This poses challenges, particularly when models have hard-to-detect failure modes and are able to take actions without oversight. In order to handle this challenge, we propose a method for a collaborative system that remains safe by having a human ultimately making decisions, while giving the model the best opportunity to convince and debate them with interpretable explanations. However, the most helpful explanation varies among individuals and may be inconsistent across stated preferences. To this end we develop an algorithm, Ardent, to efficiently learn a ranking through interaction and best assist humans complete a task. By utilising a collaborative approach, we can ensure safety and improve performance while addressing transparency and accountability concerns. Ardent enables efficient and effective decision-making by adapting to individual preferences for explanations, which we validate through extensive simulations alongside a user study involving a challenging image classification task, demonstrating consistent improvement over competing systems. △ Less

Submitted 13 November, 2023; originally announced November 2023.

Tailoring Self-Rationalizers with Multi-Reward Distillation

Authors: Sahana Ramnath, Brihi Joshi, Skyler Hallinan, Ximing Lu, Liunian Harold Li, Aaron Chan, Jack Hessel, Yejin Choi, Xiang Ren

Abstract: Large language models (LMs) are capable of generating free-text rationales to aid question answering. However, prior work 1) suggests that useful self-rationalization is emergent only at significant scales (e.g., 175B parameter GPT-3); and 2) focuses largely on downstream performance, ignoring the semantics of the rationales themselves, e.g., are they faithful, true, and helpful for humans? In thi… ▽ More Large language models (LMs) are capable of generating free-text rationales to aid question answering. However, prior work 1) suggests that useful self-rationalization is emergent only at significant scales (e.g., 175B parameter GPT-3); and 2) focuses largely on downstream performance, ignoring the semantics of the rationales themselves, e.g., are they faithful, true, and helpful for humans? In this work, we enable small-scale LMs (approx. 200x smaller than GPT-3) to generate rationales that not only improve downstream task performance, but are also more plausible, consistent, and diverse, assessed both by automatic and human evaluation. Our method, MaRio (Multi-rewArd RatIOnalization), is a multi-reward conditioned self-rationalization algorithm that optimizes multiple distinct properties like plausibility, diversity and consistency. Results on five difficult question-answering datasets StrategyQA, QuaRel, OpenBookQA, NumerSense and QASC show that not only does MaRio improve task accuracy, but it also improves the self-rationalization quality of small LMs across the aforementioned axes better than a supervised fine-tuning (SFT) baseline. Extensive human evaluations confirm that MaRio rationales are preferred vs. SFT rationales, as well as qualitative improvements in plausibility and consistency. △ Less

Submitted 22 May, 2024; v1 submitted 5 November, 2023; originally announced November 2023.

Journal ref: The Twelfth International Conference on Learning Representations, 2024

Early detection of inflammatory arthritis to improve referrals using multimodal machine learning from blood testing, semi-structured and unstructured patient records

Authors: Bing Wang, Weizi Li, Anthony Bradlow, Antoni T. Y. Chan, Eghosa Bazuaye

Abstract: Early detection of inflammatory arthritis (IA) is critical to efficient and accurate hospital referral triage for timely treatment and preventing the deterioration of the IA disease course, especially under limited healthcare resources. The manual assessment process is the most common approach in practice for the early detection of IA, but it is extremely labor-intensive and inefficient. A large a… ▽ More Early detection of inflammatory arthritis (IA) is critical to efficient and accurate hospital referral triage for timely treatment and preventing the deterioration of the IA disease course, especially under limited healthcare resources. The manual assessment process is the most common approach in practice for the early detection of IA, but it is extremely labor-intensive and inefficient. A large amount of clinical information needs to be assessed for every referral from General Practice (GP) to the hospitals. Machine learning shows great potential in automating repetitive assessment tasks and providing decision support for the early detection of IA. However, most machine learning-based methods for IA detection rely on blood testing results. But in practice, blood testing data is not always available at the point of referrals, so we need methods to leverage multimodal data such as semi-structured and unstructured data for early detection of IA. In this research, we present fusion and ensemble learning-based methods using multimodal data to assist decision-making in the early detection of IA, and a conformal prediction-based method to quantify the uncertainty of the prediction and detect any unreliable predictions. To the best of our knowledge, our study is the first attempt to utilize multimodal data to support the early detection of IA from GP referrals. △ Less

Submitted 3 November, 2023; v1 submitted 30 October, 2023; originally announced October 2023.

Comments: Accepted in The 57th Hawaii International Conference on System Sciences, 3-6 Jan 2024, Hawaii

An International Consortium for Evaluations of Societal-Scale Risks from Advanced AI

Authors: Ross Gruetzemacher, Alan Chan, Kevin Frazier, Christy Manning, Štěpán Los, James Fox, José Hernández-Orallo, John Burden, Matija Franklin, Clíodhna Ní Ghuidhir, Mark Bailey, Daniel Eth, Toby Pilditch, Kyle Kilian

Abstract: Given rapid progress toward advanced AI and risks from frontier AI systems (advanced AI systems pushing the boundaries of the AI capabilities frontier), the creation and implementation of AI governance and regulatory schemes deserves prioritization and substantial investment. However, the status quo is untenable and, frankly, dangerous. A regulatory gap has permitted AI labs to conduct research, d… ▽ More Given rapid progress toward advanced AI and risks from frontier AI systems (advanced AI systems pushing the boundaries of the AI capabilities frontier), the creation and implementation of AI governance and regulatory schemes deserves prioritization and substantial investment. However, the status quo is untenable and, frankly, dangerous. A regulatory gap has permitted AI labs to conduct research, development, and deployment activities with minimal oversight. In response, frontier AI system evaluations have been proposed as a way of assessing risks from the development and deployment of frontier AI systems. Yet, the budding AI risk evaluation ecosystem faces significant coordination challenges, such as a limited diversity of evaluators, suboptimal allocation of effort, and perverse incentives. This paper proposes a solution in the form of an international consortium for AI risk evaluations, comprising both AI developers and third-party AI risk evaluators. Such a consortium could play a critical role in international efforts to mitigate societal-scale risks from advanced AI, including in managing responsible scaling policies and coordinated evaluation-based risk response. In this paper, we discuss the current evaluation ecosystem and its shortcomings, propose an international consortium for advanced AI risk evaluations, discuss issues regarding its implementation, discuss lessons that can be learnt from previous international institutions and existing proposals for international AI governance institutions, and, finally, we recommend concrete steps to advance the establishment of the proposed consortium: (i) solicit feedback from stakeholders, (ii) conduct additional research, (iii) conduct a workshop(s) for stakeholders, (iv) analyze feedback and create final proposal, (v) solicit funding, and (vi) create a consortium. △ Less

Submitted 6 November, 2023; v1 submitted 22 October, 2023; originally announced October 2023.

Comments: 50 pages, 2 figures; updated w/ a few minor revisions based on feedback from SoLaR Workshop reviewers (on 5 page version)

Welfare Diplomacy: Benchmarking Language Model Cooperation

Authors: Gabriel Mukobi, Hannah Erlebach, Niklas Lauffer, Lewis Hammond, Alan Chan, Jesse Clifton

Abstract: The growing capabilities and increasingly widespread deployment of AI systems necessitate robust benchmarks for measuring their cooperative capabilities. Unfortunately, most multi-agent benchmarks are either zero-sum or purely cooperative, providing limited opportunities for such measurements. We introduce a general-sum variant of the zero-sum board game Diplomacy -- called Welfare Diplomacy -- in… ▽ More The growing capabilities and increasingly widespread deployment of AI systems necessitate robust benchmarks for measuring their cooperative capabilities. Unfortunately, most multi-agent benchmarks are either zero-sum or purely cooperative, providing limited opportunities for such measurements. We introduce a general-sum variant of the zero-sum board game Diplomacy -- called Welfare Diplomacy -- in which players must balance investing in military conquest and domestic welfare. We argue that Welfare Diplomacy facilitates both a clearer assessment of and stronger training incentives for cooperative capabilities. Our contributions are: (1) proposing the Welfare Diplomacy rules and implementing them via an open-source Diplomacy engine; (2) constructing baseline agents using zero-shot prompted language models; and (3) conducting experiments where we find that baselines using state-of-the-art models attain high social welfare but are exploitable. Our work aims to promote societal safety by aiding researchers in developing and assessing multi-agent AI systems. Code to evaluate Welfare Diplomacy and reproduce our experiments is available at https://github.com/mukobi/welfare-diplomacy. △ Less

Submitted 13 October, 2023; originally announced October 2023.

Unconventional Magnetic Oscillations in Kagome Mott Insulators

Authors: Guoxin Zheng, Yuan Zhu, Kuan-Wen Chen, Byungmin Kang, Dechen Zhang, Kaila Jenkins, Aaron Chan, Zhenyuan Zeng, Aini Xu, Oscar A. Valenzuela, Joanna Blawat, John Singleton, Patrick A. Lee, Shiliang Li, Lu Li

Abstract: We apply a strong magnetic field to a kagome Mott insulator with antiferromagnetic interactions which does not show magnetic ordering down to low temperatures. We observe a plateau at magnetization 1/9 Bohr magneton per magnetic ion (Cu). Furthermore, in the vicinity of this plateau we observe sets of strong oscillations in the magnetic torque, reminiscent of quantum oscillations in metals. Such o… ▽ More We apply a strong magnetic field to a kagome Mott insulator with antiferromagnetic interactions which does not show magnetic ordering down to low temperatures. We observe a plateau at magnetization 1/9 Bohr magneton per magnetic ion (Cu). Furthermore, in the vicinity of this plateau we observe sets of strong oscillations in the magnetic torque, reminiscent of quantum oscillations in metals. Such oscillations have never been seen in a wide gap insulator and point to an exotic origin. While the temperature dependence of these oscillations follows Fermi-liquid-theory predictions, they are approximately periodic in the magnetic field $H$, as opposed to $1/H$ in conventional metals. Furthermore, a strong angular dependence is observed for the period, which indicates an orbital origin for this effect. We show that the 1/9 plateau and the associated oscillations are consistent with the appearance of a quantum-spin-liquid state whose excitations are fermionic spinons that obey a Dirac spectrum. The oscillations are in response to an emergent gauge field. Our results provide strong evidence that fractionalized particles coupled to the elusive emergent gauge field have been observed. △ Less

Submitted 11 October, 2023; originally announced October 2023.

Comments: 5 pages and 3 figures in the main text, 10 additional figures in the supplement

XAI for Early Crop Classification

Authors: Ayshah Chan, Maja Schneider, Marco Körner

Abstract: We propose an approach for early crop classification through identifying important timesteps with eXplainable AI (XAI) methods. Our approach consists of training a baseline crop classification model to carry out layer-wise relevance propagation (LRP) so that the salient time step can be identified. We chose a selected number of such important time indices to create the bounding region of the short… ▽ More We propose an approach for early crop classification through identifying important timesteps with eXplainable AI (XAI) methods. Our approach consists of training a baseline crop classification model to carry out layer-wise relevance propagation (LRP) so that the salient time step can be identified. We chose a selected number of such important time indices to create the bounding region of the shortest possible classification timeframe. We identified the period 21st April 2019 to 9th August 2019 as having the best trade-off in terms of accuracy and earliness. This timeframe only suffers a 0.75% loss in accuracy as compared to using the full timeseries. We observed that the LRP-derived important timesteps also highlight small details in input values that differentiates between different classes and △ Less

Submitted 10 October, 2023; originally announced October 2023.

Resprompt: Residual Connection Prompting Advances Multi-Step Reasoning in Large Language Models

Authors: Song Jiang, Zahra Shakeri, Aaron Chan, Maziar Sanjabi, Hamed Firooz, Yinglong Xia, Bugra Akyildiz, Yizhou Sun, Jinchao Li, Qifan Wang, Asli Celikyilmaz

Abstract: Chain-of-thought (CoT) prompting, which offers step-by-step problem-solving rationales, has impressively unlocked the reasoning potential of large language models (LLMs). Yet, the standard CoT is less effective in problems demanding multiple reasoning steps. This limitation arises from the complex reasoning process in multi-step problems: later stages often depend on the results of several steps e… ▽ More Chain-of-thought (CoT) prompting, which offers step-by-step problem-solving rationales, has impressively unlocked the reasoning potential of large language models (LLMs). Yet, the standard CoT is less effective in problems demanding multiple reasoning steps. This limitation arises from the complex reasoning process in multi-step problems: later stages often depend on the results of several steps earlier, not just the results of the immediately preceding step. Such complexities suggest the reasoning process is naturally represented as a graph. The almost linear and straightforward structure of CoT prompting, however, struggles to capture this complex reasoning graph. To address this challenge, we propose Residual Connection Prompting (RESPROMPT), a new prompting strategy that advances multi-step reasoning in LLMs. Our key idea is to reconstruct the reasoning graph within prompts. We achieve this by integrating necessary connections-links present in the reasoning graph but missing in the linear CoT flow-into the prompts. Termed "residual connections", these links are pivotal in morphing the linear CoT structure into a graph representation, effectively capturing the complex reasoning graphs inherent in multi-step problems. We evaluate RESPROMPT on six benchmarks across three diverse domains: math, sequential, and commonsense reasoning. For the open-sourced LLaMA family of models, RESPROMPT yields a significant average reasoning accuracy improvement of 12.5% on LLaMA-65B and 6.8% on LLaMA2-70B. Breakdown analysis further highlights RESPROMPT particularly excels in complex multi-step reasoning: for questions demanding at least five reasoning steps, RESPROMPT outperforms the best CoT based benchmarks by a remarkable average improvement of 21.1% on LLaMA-65B and 14.3% on LLaMA2-70B. Through extensive ablation studies and analyses, we pinpoint how to most effectively build residual connections. △ Less

Submitted 8 May, 2024; v1 submitted 7 October, 2023; originally announced October 2023.

Comments: 29 pages

How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions

Authors: Lorenzo Pacchiardi, Alex J. Chan, Sören Mindermann, Ilan Moscovitz, Alexa Y. Pan, Yarin Gal, Owain Evans, Jan Brauner

Abstract: Large language models (LLMs) can "lie", which we define as outputting false statements despite "knowing" the truth in a demonstrable sense. LLMs might "lie", for example, when instructed to output misinformation. Here, we develop a simple lie detector that requires neither access to the LLM's activations (black-box) nor ground-truth knowledge of the fact in question. The detector works by asking a… ▽ More Large language models (LLMs) can "lie", which we define as outputting false statements despite "knowing" the truth in a demonstrable sense. LLMs might "lie", for example, when instructed to output misinformation. Here, we develop a simple lie detector that requires neither access to the LLM's activations (black-box) nor ground-truth knowledge of the fact in question. The detector works by asking a predefined set of unrelated follow-up questions after a suspected lie, and feeding the LLM's yes/no answers into a logistic regression classifier. Despite its simplicity, this lie detector is highly accurate and surprisingly general. When trained on examples from a single setting -- prompting GPT-3.5 to lie about factual questions -- the detector generalises out-of-distribution to (1) other LLM architectures, (2) LLMs fine-tuned to lie, (3) sycophantic lies, and (4) lies emerging in real-life scenarios such as sales. These results indicate that LLMs have distinctive lie-related behavioural patterns, consistent across architectures and contexts, which could enable general-purpose lie detection. △ Less

Submitted 26 September, 2023; originally announced September 2023.

FUTURE-AI: International consensus guideline for trustworthy and deployable artificial intelligence in healthcare

Authors: Karim Lekadir, Aasa Feragen, Abdul Joseph Fofanah, Alejandro F Frangi, Alena Buyx, Anais Emelie, Andrea Lara, Antonio R Porras, An-Wen Chan, Arcadi Navarro, Ben Glocker, Benard O Botwe, Bishesh Khanal, Brigit Beger, Carol C Wu, Celia Cintas, Curtis P Langlotz, Daniel Rueckert, Deogratias Mzurikwao, Dimitrios I Fotiadis, Doszhan Zhussupov, Enzo Ferrante, Erik Meijering, Eva Weicken, Fabio A González , et al. (95 additional authors not shown)

Abstract: Despite major advances in artificial intelligence (AI) for medicine and healthcare, the deployment and adoption of AI technologies remain limited in real-world clinical practice. In recent years, concerns have been raised about the technical, clinical, ethical and legal risks associated with medical AI. To increase real world adoption, it is essential that medical AI tools are trusted and accepted… ▽ More Despite major advances in artificial intelligence (AI) for medicine and healthcare, the deployment and adoption of AI technologies remain limited in real-world clinical practice. In recent years, concerns have been raised about the technical, clinical, ethical and legal risks associated with medical AI. To increase real world adoption, it is essential that medical AI tools are trusted and accepted by patients, clinicians, health organisations and authorities. This work describes the FUTURE-AI guideline as the first international consensus framework for guiding the development and deployment of trustworthy AI tools in healthcare. The FUTURE-AI consortium was founded in 2021 and currently comprises 118 inter-disciplinary experts from 51 countries representing all continents, including AI scientists, clinicians, ethicists, and social scientists. Over a two-year period, the consortium defined guiding principles and best practices for trustworthy AI through an iterative process comprising an in-depth literature review, a modified Delphi survey, and online consensus meetings. The FUTURE-AI framework was established based on 6 guiding principles for trustworthy AI in healthcare, i.e. Fairness, Universality, Traceability, Usability, Robustness and Explainability. Through consensus, a set of 28 best practices were defined, addressing technical, clinical, legal and socio-ethical dimensions. The recommendations cover the entire lifecycle of medical AI, from design, development and validation to regulation, deployment, and monitoring. FUTURE-AI is a risk-informed, assumption-free guideline which provides a structured approach for constructing medical AI tools that will be trusted, deployed and adopted in real-world practice. Researchers are encouraged to take the recommendations into account in proof-of-concept stages to facilitate future translation towards clinical practice of medical AI. △ Less

Submitted 8 July, 2024; v1 submitted 11 August, 2023; originally announced September 2023.

ACM Class: I.2.0; I.4.0; I.5.0

Bounds on Kemeny's constant of a graph and the Nordhaus-Gaddum problem

Authors: Sooyeong Kim, Neal Madras, Ada Chan, Mark Kempton, Stephen Kirkland, Adam Knudson

Abstract: We study Nordhaus-Gaddum problems for Kemeny's constant $\mathcal{K}(G)$ of a connected graph $G$. We prove bounds on $\min\{\mathcal{K}(G),\mathcal{K}(\overline{G})\}$ and the product $\mathcal{K}(G)\mathcal{K}(\overline{G})$ for various families of graphs. In particular, we show that if the maximum degree of a graph $G$ on $n$ vertices is $n-O(1)$ or $n-Ω(n)$, then… ▽ More We study Nordhaus-Gaddum problems for Kemeny's constant $\mathcal{K}(G)$ of a connected graph $G$. We prove bounds on $\min\{\mathcal{K}(G),\mathcal{K}(\overline{G})\}$ and the product $\mathcal{K}(G)\mathcal{K}(\overline{G})$ for various families of graphs. In particular, we show that if the maximum degree of a graph $G$ on $n$ vertices is $n-O(1)$ or $n-Ω(n)$, then $\min\{\mathcal{K}(G),\mathcal{K}(\overline{G})\}$ is at most $O(n)$. △ Less

Submitted 10 September, 2023; originally announced September 2023.

MSC Class: 05C09; 60J10; 05C81; 05C50; 05A19

Out-of-time-order correlator, many-body quantum chaos, light-like generators, and singular values

Authors: Ke Huang, Xiao Li, David A. Huse, Amos Chan

Abstract: We study out-of-time-order correlators (OTOCs) of local operators in spatial-temporal invariant or random quantum circuits using light-like generators (LLG) -- many-body operators that exist in and act along the light-like directions. We demonstrate that the OTOC can be approximated by the leading singular value of the LLG, which, for the case of generic many-body chaotic circuits, is increasingly… ▽ More We study out-of-time-order correlators (OTOCs) of local operators in spatial-temporal invariant or random quantum circuits using light-like generators (LLG) -- many-body operators that exist in and act along the light-like directions. We demonstrate that the OTOC can be approximated by the leading singular value of the LLG, which, for the case of generic many-body chaotic circuits, is increasingly accurate as the size of the LLG, $w$, increases. We analytically show that the OTOC has a decay with a universal form in the light-like direction near the causal light cone, as dictated by the sub-leading eigenvalues of LLG, $z_2$, and their degeneracies. Further, we analytically derive and numerically verify that the sub-leading eigenvalues of LLG of any size can be accessibly extracted from those of LLG of the smallest size, i.e., $z_2(w)= z_2(w=1)$. Using symmetries and recursive structures of LLG, we propose two conjectures on the universal aspects of generic many-body quantum chaotic circuits, one on the algebraic degeneracy of eigenvalues of LLG, and another on the geometric degeneracy of the sub-leading eigenvalues of LLG. As corollaries of the conjectures, we analytically derive the asymptotic form of the leading singular state, which in turn allows us to postulate and efficiently compute a product-state variational ansatz away from the asymptotic limit. We numerically test the claims with four generic circuit models of many-body quantum chaos, and contrast these statements against the cases of a dual unitary system and an integrable system. △ Less

Submitted 9 October, 2023; v1 submitted 30 August, 2023; originally announced August 2023.

Comments: 6 + 15 pages, 3 + 11 figures. Comments are welcome. Updated on 2023-10-10