-
Matrix Multiplication on Quantum Computer
Authors:
Jiaqi Yao,
Ding Liu
Abstract:
This paper introduces an innovative and practical approach to universal quantum matrix multiplication. We designed optimized quantum adders and multipliers based on Quantum Fourier Transform (QFT), which significantly reduced the number of gates used compared to classical adders and multipliers. Subsequently, we construct a basic universal quantum matrix multiplication and extend it to the Strasse…
▽ More
This paper introduces an innovative and practical approach to universal quantum matrix multiplication. We designed optimized quantum adders and multipliers based on Quantum Fourier Transform (QFT), which significantly reduced the number of gates used compared to classical adders and multipliers. Subsequently, we construct a basic universal quantum matrix multiplication and extend it to the Strassen algorithm. We conduct comparative experiments to analyze the performance of the quantum matrix multiplication and evaluate the acceleration provided by the optimized quantum adder and multiplier. Furthermore, we investigate the advantages and disadvantages of the quantum Strassen algorithm compared to basic quantum matrix multiplication.
△ Less
Submitted 6 August, 2024;
originally announced August 2024.
-
Subspace-projected multireference covariant density functional theory
Authors:
X. Zhang,
C. C. Wang,
C. R. Ding,
J. M. Yao
Abstract:
Multireference density functional theory (MR-DFT) has been a pivotal method for studying nuclear spectroscopy and neutrinoless double-beta ($0νββ$) decay. However, quantifying their theoretical uncertainties has been a significant challenge due to the computational demands. This study introduces a subspace-projected covariant density functional theory (SP-CDFT), which efficiently emulates MR-CDFT…
▽ More
Multireference density functional theory (MR-DFT) has been a pivotal method for studying nuclear spectroscopy and neutrinoless double-beta ($0νββ$) decay. However, quantifying their theoretical uncertainties has been a significant challenge due to the computational demands. This study introduces a subspace-projected covariant density functional theory (SP-CDFT), which efficiently emulates MR-CDFT calculations for nuclear low-lying states. This approach leverages the eigenvector continuation method combined with the quantum-number projected generator coordinate method, based on a relativistic energy density functional (EDF). We apply SP-CDFT to investigate the correlations among the physical quantities of nuclear matter, nuclear low-lying spectroscopy, and the nuclear matrix elements (NMEs) of $0νββ$ decay in the two heaviest candidate nuclei. Our findings reveal generally strong correlations between the NMEs of $0νββ$ decay and the excitation energy of the $2_1^+$ state, as well as the $E2$ transition strength, although these correlations vary significantly among nuclei. The Bayesian analysis, which integrates the properties of nuclear matter and low-lying states, yields mean values and statistical uncertainties for the NMEs 4.33(5) for $^{136}$Xe and 5.51(14) for $^{150}$Nd. This work also paves the way for refining nuclear EDF parameters using spectroscopic data.
△ Less
Submitted 1 August, 2024;
originally announced August 2024.
-
An Application of Large Language Models to Coding Negotiation Transcripts
Authors:
Ray Friedman,
Jaewoo Cho,
Jeanne Brett,
Xuhui Zhan,
Ningyu Han,
Sriram Kannan,
Yingxiang Ma,
Jesse Spencer-Smith,
Elisabeth Jäckel,
Alfred Zerres,
Madison Hooper,
Katie Babbit,
Manish Acharya,
Wendi Adair,
Soroush Aslani,
Tayfun Aykaç,
Chris Bauman,
Rebecca Bennett,
Garrett Brady,
Peggy Briggs,
Cheryl Dowie,
Chase Eck,
Igmar Geiger,
Frank Jacob,
Molly Kern
, et al. (33 additional authors not shown)
Abstract:
In recent years, Large Language Models (LLM) have demonstrated impressive capabilities in the field of natural language processing (NLP). This paper explores the application of LLMs in negotiation transcript analysis by the Vanderbilt AI Negotiation Lab. Starting in September 2022, we applied multiple strategies using LLMs from zero shot learning to fine tuning models to in-context learning). The…
▽ More
In recent years, Large Language Models (LLM) have demonstrated impressive capabilities in the field of natural language processing (NLP). This paper explores the application of LLMs in negotiation transcript analysis by the Vanderbilt AI Negotiation Lab. Starting in September 2022, we applied multiple strategies using LLMs from zero shot learning to fine tuning models to in-context learning). The final strategy we developed is explained, along with how to access and use the model. This study provides a sense of both the opportunities and roadblocks for the implementation of LLMs in real life applications and offers a model for how LLMs can be applied to coding in other fields.
△ Less
Submitted 18 July, 2024;
originally announced July 2024.
-
Unstable Invariant Measures and Connecting Orbits of Cooperative McKean-Vlasov SDEs
Authors:
Chunlin Liu,
Baoyou Qu,
Jinxiang Yao,
Yanpeng Zhi
Abstract:
A general framework for studying McKean-Vlasov SDEs via monotone dynamical systems is established in this paper. Under a cooperative condition, we show McKean-Vlasov SDEs admit a comparison principle with respect to the stochastic order, and generate monotone dynamical systems on the $2$-Wasserstein space. Our main results prove the existence of unstable invariant measures, total orderedness of in…
▽ More
A general framework for studying McKean-Vlasov SDEs via monotone dynamical systems is established in this paper. Under a cooperative condition, we show McKean-Vlasov SDEs admit a comparison principle with respect to the stochastic order, and generate monotone dynamical systems on the $2$-Wasserstein space. Our main results prove the existence of unstable invariant measures, total orderedness of invariant measures, and the existence of monotone connecting orbits between order-related invariant measures for general cooperative McKean-Vlasov SDEs. To achieve our goals, we adopt the theory of monotone dynamical systems, extend the connecting orbit theorem, and deduce a dichotomy structure of equilibria. This method is different from existing approaches, like propagation of chaos and Fokker-Planck equations. A wide range of classical examples are covered by our framework, such as granular media equations in double-well and multi-well confinement potentials with quadratic interaction, double-well landscapes with perturbation, and higher dimensional equations, even driven by multiplicative noises.
△ Less
Submitted 26 July, 2024;
originally announced July 2024.
-
Know Your Limits: A Survey of Abstention in Large Language Models
Authors:
Bingbing Wen,
Jihan Yao,
Shangbin Feng,
Chenjun Xu,
Yulia Tsvetkov,
Bill Howe,
Lucy Lu Wang
Abstract:
Abstention, the refusal of large language models (LLMs) to provide an answer, is increasingly recognized for its potential to mitigate hallucinations and enhance safety in LLM systems. In this survey, we introduce a framework to examine abstention from three perspectives: the query, the model, and human values. We organize the literature on abstention methods, benchmarks, and evaluation metrics us…
▽ More
Abstention, the refusal of large language models (LLMs) to provide an answer, is increasingly recognized for its potential to mitigate hallucinations and enhance safety in LLM systems. In this survey, we introduce a framework to examine abstention from three perspectives: the query, the model, and human values. We organize the literature on abstention methods, benchmarks, and evaluation metrics using this framework, and discuss merits and limitations of prior work. We further identify and motivate areas for future work, centered around whether abstention can be achieved as a meta-capability that transcends specific tasks or domains, while still providing opportunities to optimize abstention abilities based on context.
△ Less
Submitted 8 August, 2024; v1 submitted 25 July, 2024;
originally announced July 2024.
-
LKCell: Efficient Cell Nuclei Instance Segmentation with Large Convolution Kernels
Authors:
Ziwei Cui,
Jingfeng Yao,
Lunbin Zeng,
Juan Yang,
Wenyu Liu,
Xinggang Wang
Abstract:
The segmentation of cell nuclei in tissue images stained with the blood dye hematoxylin and eosin (H$\&$E) is essential for various clinical applications and analyses. Due to the complex characteristics of cellular morphology, a large receptive field is considered crucial for generating high-quality segmentation. However, previous methods face challenges in achieving a balance between the receptiv…
▽ More
The segmentation of cell nuclei in tissue images stained with the blood dye hematoxylin and eosin (H$\&$E) is essential for various clinical applications and analyses. Due to the complex characteristics of cellular morphology, a large receptive field is considered crucial for generating high-quality segmentation. However, previous methods face challenges in achieving a balance between the receptive field and computational burden. To address this issue, we propose LKCell, a high-accuracy and efficient cell segmentation method. Its core insight lies in unleashing the potential of large convolution kernels to achieve computationally efficient large receptive fields. Specifically, (1) We transfer pre-trained large convolution kernel models to the medical domain for the first time, demonstrating their effectiveness in cell segmentation. (2) We analyze the redundancy of previous methods and design a new segmentation decoder based on large convolution kernels. It achieves higher performance while significantly reducing the number of parameters. We evaluate our method on the most challenging benchmark and achieve state-of-the-art results (0.5080 mPQ) in cell nuclei instance segmentation with only 21.6% FLOPs compared with the previous leading method. Our source code and models are available at https://github.com/hustvl/LKCell.
△ Less
Submitted 25 July, 2024;
originally announced July 2024.
-
SoNIC: Safe Social Navigation with Adaptive Conformal Inference and Constrained Reinforcement Learning
Authors:
Jianpeng Yao,
Xiaopan Zhang,
Yu Xia,
Zejin Wang,
Amit K. Roy-Chowdhury,
Jiachen Li
Abstract:
Reinforcement Learning (RL) has enabled social robots to generate trajectories without human-designed rules or interventions, which makes it more effective than hard-coded systems for generalizing to complex real-world scenarios. However, social navigation is a safety-critical task that requires robots to avoid collisions with pedestrians while previous RL-based solutions fall short in safety perf…
▽ More
Reinforcement Learning (RL) has enabled social robots to generate trajectories without human-designed rules or interventions, which makes it more effective than hard-coded systems for generalizing to complex real-world scenarios. However, social navigation is a safety-critical task that requires robots to avoid collisions with pedestrians while previous RL-based solutions fall short in safety performance in complex environments. To enhance the safety of RL policies, to the best of our knowledge, we propose the first algorithm, SoNIC, that integrates adaptive conformal inference (ACI) with constrained reinforcement learning (CRL) to learn safe policies for social navigation. More specifically, our method augments RL observations with ACI-generated nonconformity scores and provides explicit guidance for agents to leverage the uncertainty metrics to avoid safety-critical areas by incorporating safety constraints with spatial relaxation. Our method outperforms state-of-the-art baselines in terms of both safety and adherence to social norms by a large margin and demonstrates much stronger robustness to out-of-distribution scenarios. Our code and video demos are available on our project website: https://sonic-social-nav.github.io/.
△ Less
Submitted 24 July, 2024;
originally announced July 2024.
-
Blind Beamforming for Coverage Enhancement with Intelligent Reflecting Surface
Authors:
Fan Xu,
Jiawei Yao,
Wenhai Lai,
Kaiming Shen,
Xin Li,
Xin Chen,
Zhi-Quan Luo
Abstract:
Conventional policy for configuring an intelligent reflecting surface (IRS) typically requires channel state information (CSI), thus incurring substantial overhead costs and facing incompatibility with the current network protocols. This paper proposes a blind beamforming strategy in the absence of CSI, aiming to boost the minimum signal-to-noise ratio (SNR) among all the receiver positions, namel…
▽ More
Conventional policy for configuring an intelligent reflecting surface (IRS) typically requires channel state information (CSI), thus incurring substantial overhead costs and facing incompatibility with the current network protocols. This paper proposes a blind beamforming strategy in the absence of CSI, aiming to boost the minimum signal-to-noise ratio (SNR) among all the receiver positions, namely the coverage enhancement. Although some existing works already consider the IRS-assisted coverage enhancement without CSI, they assume certain position-channel models through which the channels can be recovered from the geographic locations. In contrast, our approach solely relies on the received signal power data, not assuming any position-channel model. We examine the achievability and converse of the proposed blind beamforming method. If the IRS has $N$ reflective elements and there are $U$ receiver positions, then our method guarantees the minimum SNR of $Ω(N^2/U)$ -- which is fairly close to the upper bound $O(N+N^2\sqrt{\ln (NU)}/\sqrt[4]{U})$. Aside from the simulation results, we justify the practical use of blind beamforming in a field test at 2.6 GHz. According to the real-world experiment, the proposed blind beamforming method boosts the minimum SNR across seven random positions in a conference room by 18.22 dB, while the position-based method yields a boost of 12.08 dB.
△ Less
Submitted 17 July, 2024;
originally announced July 2024.
-
Facial Affect Recognition based on Multi Architecture Encoder and Feature Fusion for the ABAW7 Challenge
Authors:
Kang Shen,
Xuxiong Liu,
Boyan Wang,
Jun Yao,
Xin Liu,
Yujie Guan,
Yu Wang,
Gengchen Li,
Xiao Sun
Abstract:
In this paper, we present our approach to addressing the challenges of the 7th ABAW competition. The competition comprises three sub-challenges: Valence Arousal (VA) estimation, Expression (Expr) classification, and Action Unit (AU) detection. To tackle these challenges, we employ state-of-the-art models to extract powerful visual features. Subsequently, a Transformer Encoder is utilized to integr…
▽ More
In this paper, we present our approach to addressing the challenges of the 7th ABAW competition. The competition comprises three sub-challenges: Valence Arousal (VA) estimation, Expression (Expr) classification, and Action Unit (AU) detection. To tackle these challenges, we employ state-of-the-art models to extract powerful visual features. Subsequently, a Transformer Encoder is utilized to integrate these features for the VA, Expr, and AU sub-challenges. To mitigate the impact of varying feature dimensions, we introduce an affine module to align the features to a common dimension. Overall, our results significantly outperform the baselines.
△ Less
Submitted 26 July, 2024; v1 submitted 16 July, 2024;
originally announced July 2024.
-
Compound Expression Recognition via Multi Model Ensemble for the ABAW7 Challenge
Authors:
Xuxiong Liu,
Kang Shen,
Jun Yao,
Boyan Wang,
Minrui Liu,
Liuwei An,
Zishun Cui,
Weijie Feng,
Xiao Sun
Abstract:
Compound Expression Recognition (CER) is vital for effective interpersonal interactions. Human emotional expressions are inherently complex due to the presence of compound expressions, requiring the consideration of both local and global facial cues for accurate judgment. In this paper, we propose an ensemble learning-based solution to address this complexity. Our approach involves training three…
▽ More
Compound Expression Recognition (CER) is vital for effective interpersonal interactions. Human emotional expressions are inherently complex due to the presence of compound expressions, requiring the consideration of both local and global facial cues for accurate judgment. In this paper, we propose an ensemble learning-based solution to address this complexity. Our approach involves training three distinct expression classification models using convolutional networks, Vision Transformers, and multiscale local attention networks. By employing late fusion for model ensemble, we combine the outputs of these models to predict the final results. Our method demonstrates high accuracy on the RAF-DB datasets and is capable of recognizing expressions in certain portions of the C-EXPR-DB through zero-shot learning.
△ Less
Submitted 26 July, 2024; v1 submitted 16 July, 2024;
originally announced July 2024.
-
MUSA: Multi-lingual Speaker Anonymization via Serial Disentanglement
Authors:
Jixun Yao,
Qing Wang,
Pengcheng Guo,
Ziqian Ning,
Yuguang Yang,
Yu Pan,
Lei Xie
Abstract:
Speaker anonymization is an effective privacy protection solution designed to conceal the speaker's identity while preserving the linguistic content and para-linguistic information of the original speech. While most prior studies focus solely on a single language, an ideal speaker anonymization system should be capable of handling multiple languages. This paper proposes MUSA, a Multi-lingual Speak…
▽ More
Speaker anonymization is an effective privacy protection solution designed to conceal the speaker's identity while preserving the linguistic content and para-linguistic information of the original speech. While most prior studies focus solely on a single language, an ideal speaker anonymization system should be capable of handling multiple languages. This paper proposes MUSA, a Multi-lingual Speaker Anonymization approach that employs a serial disentanglement strategy to perform a step-by-step disentanglement from a global time-invariant representation to a temporal time-variant representation. By utilizing semantic distillation and self-supervised speaker distillation, the serial disentanglement strategy can avoid strong inductive biases and exhibit superior generalization performance across different languages. Meanwhile, we propose a straightforward anonymization strategy that employs empty embedding with zero values to simulate the speaker identity concealment process, eliminating the need for conversion to a pseudo-speaker identity and thereby reducing the complexity of speaker anonymization process. Experimental results on VoicePrivacy official datasets and multi-lingual datasets demonstrate that MUSA can effectively protect speaker privacy while preserving linguistic content and para-linguistic information.
△ Less
Submitted 16 July, 2024;
originally announced July 2024.
-
Fluid Antenna-Assisted Simultaneous Wireless Information and Power Transfer Systems
Authors:
Liaoshi Zhou,
Junteng Yao,
Tuo Wu,
Ming Jin,
Chau Yuen,
Fumiyuki Adachi
Abstract:
This paper examines a fluid antenna (FA)-assisted simultaneous wireless information and power transfer (SWIPT) system. Unlike traditional SWIPT systems with fixed-position antennas (FPAs), our FA-assisted system enables dynamic reconfiguration of the radio propagation environment by adjusting the positions of FAs. This capability enhances both energy harvesting and communication performance. The s…
▽ More
This paper examines a fluid antenna (FA)-assisted simultaneous wireless information and power transfer (SWIPT) system. Unlike traditional SWIPT systems with fixed-position antennas (FPAs), our FA-assisted system enables dynamic reconfiguration of the radio propagation environment by adjusting the positions of FAs. This capability enhances both energy harvesting and communication performance. The system comprises a base station (BS) equipped with multiple FAs that transmit signals to an energy receiver (ER) and an information receiver (IR), both equipped with a single FA. Our objective is to maximize the communication rate between the BS and the IR while satisfying the harvested power requirement of the ER. This involves jointly optimizing the BS's transmit beamforming and the positions of all FAs. To address this complex convex optimization problem, we employ an alternating optimization (AO) approach, decomposing it into three sub-problems and solving them iteratively using first and second-order Taylor expansions. Simulation results validate the effectiveness of our proposed FA-assisted SWIPT system, demonstrating significant performance improvements over traditional FPA-based systems.
△ Less
Submitted 23 July, 2024; v1 submitted 15 July, 2024;
originally announced July 2024.
-
CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responses
Authors:
Jing Yao,
Xiaoyuan Yi,
Xing Xie
Abstract:
The rapid progress in Large Language Models (LLMs) poses potential risks such as generating unethical content. Assessing LLMs' values can help expose their misalignment, but relies on reference-free evaluators, e.g., fine-tuned LLMs or close-source ones like GPT-4, to identify values reflected in generated responses. Nevertheless, these evaluators face two challenges in open-ended value evaluation…
▽ More
The rapid progress in Large Language Models (LLMs) poses potential risks such as generating unethical content. Assessing LLMs' values can help expose their misalignment, but relies on reference-free evaluators, e.g., fine-tuned LLMs or close-source ones like GPT-4, to identify values reflected in generated responses. Nevertheless, these evaluators face two challenges in open-ended value evaluation: they should align with changing human value definitions with minimal annotation, against their own bias (adaptability), and detect varying value expressions and scenarios robustly (generalizability). To handle these challenges, we introduce CLAVE, a novel framework which integrates two complementary LLMs, a large one to extract high-level value concepts from a few human labels, leveraging its extensive knowledge and generalizability, and a smaller one fine-tuned on such concepts to better align with human value understanding. This dual-model approach enables calibration with any value systems using <100 human-labeled samples per value type. Then we present ValEval, a comprehensive dataset comprising 13k+ (text,value,label) tuples across diverse domains, covering three major value systems. We benchmark the capabilities of 12+ popular LLM evaluators and analyze their strengths and weaknesses. Our findings reveal that combining fine-tuned small models and prompt-based large ones serves as a superior balance in value evaluation.
△ Less
Submitted 15 July, 2024;
originally announced July 2024.
-
A Framework of FAS-RIS Systems: Performance Analysis and Throughput Optimization
Authors:
Junteng Yao,
Xiazhi Lai,
Kangda Zhi,
Tuo Wu,
Ming Jin,
Cunhua Pan,
Maged Elkashlan,
Chau Yuen,
Kai-Kit Wong
Abstract:
In this paper, we investigate reconfigurable intelligent surface (RIS)-assisted communication systems which involve a fixed-antenna base station (BS) and a mobile user (MU) that is equipped with fluid antenna system (FAS). Specifically, the RIS is utilized to enable communication for the user whose direct link from the base station is blocked by obstacles. We propose a comprehensive framework that…
▽ More
In this paper, we investigate reconfigurable intelligent surface (RIS)-assisted communication systems which involve a fixed-antenna base station (BS) and a mobile user (MU) that is equipped with fluid antenna system (FAS). Specifically, the RIS is utilized to enable communication for the user whose direct link from the base station is blocked by obstacles. We propose a comprehensive framework that provides transmission design for both static scenarios with the knowledge of channel state information (CSI) and harsh environments where CSI is hard to acquire. It leads to two approaches: a CSI-based scheme where CSI is available, and a CSI-free scheme when CSI is inaccessible. Given the complex spatial correlations in FAS, we employ block-diagonal matrix approximation and independent antenna equivalent models to simplify the derivation of outage probabilities in both cases. Based on the derived outage probabilities, we then optimize the throughput of the FAS-RIS system. For the CSI-based scheme, we first propose a gradient ascent-based algorithm to obtain a near-optimal solution. Then, to address the possible high computational complexity in the gradient algorithm, we approximate the objective function and confirm a unique optimal solution accessible through a bisection search method. For the CSI-free scheme, we apply the partial gradient ascent algorithm, reducing complexity further than full gradient algorithms. We also approximate the objective function and derive a locally optimal closed-form solution to maximize throughput. Simulation results validate the effectiveness of the proposed framework for the transmission design in FAS-RIS systems.
△ Less
Submitted 10 July, 2024;
originally announced July 2024.
-
Reprogramming Distillation for Medical Foundation Models
Authors:
Yuhang Zhou,
Siyuan Du,
Haolin Li,
Jiangchao Yao,
Ya Zhang,
Yanfeng Wang
Abstract:
Medical foundation models pre-trained on large-scale datasets have demonstrated powerful versatile capabilities for various tasks. However, due to the gap between pre-training tasks (or modalities) and downstream tasks (or modalities), the real-world computation and speed constraints, it might not be straightforward to apply medical foundation models in the downstream scenarios. Previous methods,…
▽ More
Medical foundation models pre-trained on large-scale datasets have demonstrated powerful versatile capabilities for various tasks. However, due to the gap between pre-training tasks (or modalities) and downstream tasks (or modalities), the real-world computation and speed constraints, it might not be straightforward to apply medical foundation models in the downstream scenarios. Previous methods, such as parameter efficient fine-tuning (PEFT) methods and knowledge distillation (KD) methods, are unable to simultaneously address the task (or modality) inconsistency and achieve personalized lightweight deployment under diverse real-world demands. To address the above issues, we propose a novel framework called Reprogramming Distillation (RD). On one hand, RD reprograms the original feature space of the foundation model so that it is more relevant to downstream scenarios, aligning tasks and modalities. On the other hand, through a co-training mechanism and a shared classifier, connections are established between the reprogrammed knowledge and the knowledge of student models, ensuring that the reprogrammed feature space can be smoothly mimic by the student model of different structures. Further, to reduce the randomness under different training conditions, we design a Centered Kernel Alignment (CKA) distillation to promote robust knowledge transfer. Empirically, we show that on extensive datasets, RD consistently achieve superior performance compared with previous PEFT and KD methods.
△ Less
Submitted 8 July, 2024;
originally announced July 2024.
-
Audio-driven High-resolution Seamless Talking Head Video Editing via StyleGAN
Authors:
Jiacheng Su,
Kunhong Liu,
Liyan Chen,
Junfeng Yao,
Qingsong Liu,
Dongdong Lv
Abstract:
The existing methods for audio-driven talking head video editing have the limitations of poor visual effects. This paper tries to tackle this problem through editing talking face images seamless with different emotions based on two modules: (1) an audio-to-landmark module, consisting of the CrossReconstructed Emotion Disentanglement and an alignment network module. It bridges the gap between speec…
▽ More
The existing methods for audio-driven talking head video editing have the limitations of poor visual effects. This paper tries to tackle this problem through editing talking face images seamless with different emotions based on two modules: (1) an audio-to-landmark module, consisting of the CrossReconstructed Emotion Disentanglement and an alignment network module. It bridges the gap between speech and facial motions by predicting corresponding emotional landmarks from speech; (2) a landmark-based editing module edits face videos via StyleGAN. It aims to generate the seamless edited video consisting of the emotion and content components from the input audio. Extensive experiments confirm that compared with state-of-the-arts methods, our method provides high-resolution videos with high visual quality.
△ Less
Submitted 7 July, 2024;
originally announced July 2024.
-
SurgicalGaussian: Deformable 3D Gaussians for High-Fidelity Surgical Scene Reconstruction
Authors:
Weixing Xie,
Junfeng Yao,
Xianpeng Cao,
Qiqin Lin,
Zerui Tang,
Xiao Dong,
Xiaohu Guo
Abstract:
Dynamic reconstruction of deformable tissues in endoscopic video is a key technology for robot-assisted surgery. Recent reconstruction methods based on neural radiance fields (NeRFs) have achieved remarkable results in the reconstruction of surgical scenes. However, based on implicit representation, NeRFs struggle to capture the intricate details of objects in the scene and cannot achieve real-tim…
▽ More
Dynamic reconstruction of deformable tissues in endoscopic video is a key technology for robot-assisted surgery. Recent reconstruction methods based on neural radiance fields (NeRFs) have achieved remarkable results in the reconstruction of surgical scenes. However, based on implicit representation, NeRFs struggle to capture the intricate details of objects in the scene and cannot achieve real-time rendering. In addition, restricted single view perception and occluded instruments also propose special challenges in surgical scene reconstruction. To address these issues, we develop SurgicalGaussian, a deformable 3D Gaussian Splatting method to model dynamic surgical scenes. Our approach models the spatio-temporal features of soft tissues at each time stamp via a forward-mapping deformation MLP and regularization to constrain local 3D Gaussians to comply with consistent movement. With the depth initialization strategy and tool mask-guided training, our method can remove surgical instruments and reconstruct high-fidelity surgical scenes. Through experiments on various surgical videos, our network outperforms existing method on many aspects, including rendering quality, rendering speed and GPU usage. The project page can be found at https://surgicalgaussian.github.io.
△ Less
Submitted 6 July, 2024;
originally announced July 2024.
-
ASPS: Augmented Segment Anything Model for Polyp Segmentation
Authors:
Huiqian Li,
Dingwen Zhang,
Jieru Yao,
Longfei Han,
Zhongyu Li,
Junwei Han
Abstract:
Polyp segmentation plays a pivotal role in colorectal cancer diagnosis. Recently, the emergence of the Segment Anything Model (SAM) has introduced unprecedented potential for polyp segmentation, leveraging its powerful pre-training capability on large-scale datasets. However, due to the domain gap between natural and endoscopy images, SAM encounters two limitations in achieving effective performan…
▽ More
Polyp segmentation plays a pivotal role in colorectal cancer diagnosis. Recently, the emergence of the Segment Anything Model (SAM) has introduced unprecedented potential for polyp segmentation, leveraging its powerful pre-training capability on large-scale datasets. However, due to the domain gap between natural and endoscopy images, SAM encounters two limitations in achieving effective performance in polyp segmentation. Firstly, its Transformer-based structure prioritizes global and low-frequency information, potentially overlooking local details, and introducing bias into the learned features. Secondly, when applied to endoscopy images, its poor out-of-distribution (OOD) performance results in substandard predictions and biased confidence output. To tackle these challenges, we introduce a novel approach named Augmented SAM for Polyp Segmentation (ASPS), equipped with two modules: Cross-branch Feature Augmentation (CFA) and Uncertainty-guided Prediction Regularization (UPR). CFA integrates a trainable CNN encoder branch with a frozen ViT encoder, enabling the integration of domain-specific knowledge while enhancing local features and high-frequency details. Moreover, UPR ingeniously leverages SAM's IoU score to mitigate uncertainty during the training procedure, thereby improving OOD performance and domain generalization. Extensive experimental results demonstrate the effectiveness and utility of the proposed method in improving SAM's performance in polyp segmentation. Our code is available at https://github.com/HuiqianLi/ASPS.
△ Less
Submitted 30 June, 2024;
originally announced July 2024.
-
Dual-view Aware Smart Contract Vulnerability Detection for Ethereum
Authors:
Jiacheng Yao,
Maolin Wang,
Wanqi Chen,
Chengxiang Jin,
Jiajun Zhou,
Shanqing Yu,
Qi Xuan
Abstract:
The wide application of Ethereum technology has brought technological innovation to traditional industries. As one of Ethereum's core applications, smart contracts utilize diverse contract codes to meet various functional needs and have gained widespread use. However, the non-tamperability of smart contracts, coupled with vulnerabilities caused by natural flaws or human errors, has brought unprece…
▽ More
The wide application of Ethereum technology has brought technological innovation to traditional industries. As one of Ethereum's core applications, smart contracts utilize diverse contract codes to meet various functional needs and have gained widespread use. However, the non-tamperability of smart contracts, coupled with vulnerabilities caused by natural flaws or human errors, has brought unprecedented challenges to blockchain security. Therefore, in order to ensure the healthy development of blockchain technology and the stability of the blockchain community, it is particularly important to study the vulnerability detection techniques for smart contracts. In this paper, we propose a Dual-view Aware Smart Contract Vulnerability Detection Framework named DVDet. The framework initially converts the source code and bytecode of smart contracts into weighted graphs and control flow sequences, capturing potential risk features from these two perspectives and integrating them for analysis, ultimately achieving effective contract vulnerability detection. Comprehensive experiments on the Ethereum dataset show that our method outperforms others in detecting vulnerabilities.
△ Less
Submitted 29 June, 2024;
originally announced July 2024.
-
Timing and Scintillation Studies of Pulsars in Globular Cluster M3 (NGC 5272) with FAST
Authors:
Baoda Li,
Li-yun Zhang,
Jumei Yao,
Dejiang Yin,
Ralph P. Eatough,
Minghui Li,
Yifeng Li,
Yujie Lian,
Yu Pan,
Yinfeng Dai,
Yaowei Li,
Xingnan Zhang,
Tianhao Su,
Yuxiao Wu,
Tong Liu,
Kuo Liu,
Lin Wang,
Lei Qian,
Zhichen Pan
Abstract:
We present the phase-connected timing solutions of all the five pulsars in globular cluster (GC) M3 (NGC 5272), namely PSRs M3A to F (PSRs J1342+2822A to F), with the exception of PSR M3C, from FAST archival data. In these timing solutions, those of PSRs M3E, and F are obtained for the first time. We find that PSRs M3E and F have low mass companions, and are in circular orbits with periods of 7.1…
▽ More
We present the phase-connected timing solutions of all the five pulsars in globular cluster (GC) M3 (NGC 5272), namely PSRs M3A to F (PSRs J1342+2822A to F), with the exception of PSR M3C, from FAST archival data. In these timing solutions, those of PSRs M3E, and F are obtained for the first time. We find that PSRs M3E and F have low mass companions, and are in circular orbits with periods of 7.1 and 3.0 days, respectively. For PSR M3C, we have not detected it in all the 41 observations. We found no X-ray counterparts for these pulsars in archival Chandra images in the band of 0.2-20 keV. We noticed that the pulsars in M3 seem to be native. From the Auto-Correlation Function (ACF) analysis of the M3A's and M3B's dynamic spectra, the scintillation timescale ranges from $7.0\pm0.3$ min to $60.0\pm0.6$ min, and the scintillation bandwidth ranges from $4.6\pm0.2$ MHz to $57.1\pm1.1$ MHz. The measured scintillation bandwidths from the dynamic spectra indicate strong scintillation, and the scattering medium is anisotropic. From the secondary spectra, we captured a scintillation arc only for PSR M3B with a curvature of $649\pm23 {\rm m}^{-1} {\rm mHz}^{-2}$.
△ Less
Submitted 26 June, 2024;
originally announced June 2024.
-
PVUW 2024 Challenge on Complex Video Understanding: Methods and Results
Authors:
Henghui Ding,
Chang Liu,
Yunchao Wei,
Nikhila Ravi,
Shuting He,
Song Bai,
Philip Torr,
Deshui Miao,
Xin Li,
Zhenyu He,
Yaowei Wang,
Ming-Hsuan Yang,
Zhensong Xu,
Jiangtao Yao,
Chengjing Wu,
Ting Liu,
Luoqi Liu,
Xinyu Liu,
Jing Zhang,
Kexin Zhang,
Yuting Yang,
Licheng Jiao,
Shuyuan Yang,
Mingqi Gao,
Jingnan Luo
, et al. (12 additional authors not shown)
Abstract:
Pixel-level Video Understanding in the Wild Challenge (PVUW) focus on complex video understanding. In this CVPR 2024 workshop, we add two new tracks, Complex Video Object Segmentation Track based on MOSE dataset and Motion Expression guided Video Segmentation track based on MeViS dataset. In the two new tracks, we provide additional videos and annotations that feature challenging elements, such as…
▽ More
Pixel-level Video Understanding in the Wild Challenge (PVUW) focus on complex video understanding. In this CVPR 2024 workshop, we add two new tracks, Complex Video Object Segmentation Track based on MOSE dataset and Motion Expression guided Video Segmentation track based on MeViS dataset. In the two new tracks, we provide additional videos and annotations that feature challenging elements, such as the disappearance and reappearance of objects, inconspicuous small objects, heavy occlusions, and crowded environments in MOSE. Moreover, we provide a new motion expression guided video segmentation dataset MeViS to study the natural language-guided video understanding in complex environments. These new videos, sentences, and annotations enable us to foster the development of a more comprehensive and robust pixel-level understanding of video scenes in complex environments and realistic scenarios. The MOSE challenge had 140 registered teams in total, 65 teams participated the validation phase and 12 teams made valid submissions in the final challenge phase. The MeViS challenge had 225 registered teams in total, 50 teams participated the validation phase and 5 teams made valid submissions in the final challenge phase.
△ Less
Submitted 24 June, 2024;
originally announced June 2024.
-
Near-Field Mobile Tracking: A Framework of Using XL-RIS Information
Authors:
Tuo Wu,
Cunhua Pan,
Kangda Zhi,
Junteng Yao,
Hong Ren,
Maged Elkashlan,
Chau Yuen
Abstract:
This paper introduces a novel mobile tracking framework leveraging the high-dimensional signal received from extremely large-scale (XL) reconfigurable intelligent surfaces (RIS). This received signal, named XL-RIS information, has a much larger data dimension and therefore offers a richer feature set compared to the traditional base station (BS) received signal, i.e., BS information, enabling more…
▽ More
This paper introduces a novel mobile tracking framework leveraging the high-dimensional signal received from extremely large-scale (XL) reconfigurable intelligent surfaces (RIS). This received signal, named XL-RIS information, has a much larger data dimension and therefore offers a richer feature set compared to the traditional base station (BS) received signal, i.e., BS information, enabling more accurate tracking of mobile users (MUs). As the first step, we present an XL-RIS information reconstruction (XL-RIS-IR) algorithm to reconstruct the high-dimensional XL-RIS information from the low-dimensional BS information. Building on this, this paper proposes a comprehensive framework for mobile tracking, consisting of a Feature Extraction Module and a Mobile Tracking Module. The Feature Extraction Module incorporates a convolutional neural network (CNN) extractor for spatial features, a time and frequency (T$\&$F) extractor for domain features, and a near-field angles of arrival (AoAs) extractor for capturing AoA features within the XL-RIS. These features are combined into a comprehensive feature vector, forming a time-varying sequence fed into the Mobile Tracking Module, which employs an Auto-encoder (AE) with a stacked bidirectional long short-term memory (Bi-LSTM) encoder and a standard LSTM decoder to predict MUs' positions in the upcoming time slot. Simulation results confirm that the tracking accuracy of our proposed framework is significantly enhanced by using reconstructed XL-RIS information and exhibits substantial robustness to signal-to-noise ratio (SNR) variations.
△ Less
Submitted 5 August, 2024; v1 submitted 3 April, 2024;
originally announced June 2024.
-
Towards Better Graph-based Cross-document Relation Extraction via Non-bridge Entity Enhancement and Prediction Debiasing
Authors:
Hao Yue,
Shaopeng Lai,
Chengyi Yang,
Liang Zhang,
Junfeng Yao,
Jinsong Su
Abstract:
Cross-document Relation Extraction aims to predict the relation between target entities located in different documents. In this regard, the dominant models commonly retain useful information for relation prediction via bridge entities, which allows the model to elaborately capture the intrinsic interdependence between target entities. However, these studies ignore the non-bridge entities, each of…
▽ More
Cross-document Relation Extraction aims to predict the relation between target entities located in different documents. In this regard, the dominant models commonly retain useful information for relation prediction via bridge entities, which allows the model to elaborately capture the intrinsic interdependence between target entities. However, these studies ignore the non-bridge entities, each of which co-occurs with only one target entity and offers the semantic association between target entities for relation prediction. Besides, the commonly-used dataset--CodRED contains substantial NA instances, leading to the prediction bias during inference. To address these issues, in this paper, we propose a novel graph-based cross-document RE model with non-bridge entity enhancement and prediction debiasing. Specifically, we use a unified entity graph to integrate numerous non-bridge entities with target entities and bridge entities, modeling various associations between them, and then use a graph recurrent network to encode this graph. Finally, we introduce a novel debiasing strategy to calibrate the original prediction distribution. Experimental results on the closed and open settings show that our model significantly outperforms all baselines, including the GPT-3.5-turbo and InstructUIE, achieving state-of-the-art performance. Particularly, our model obtains 66.23% and 55.87% AUC points in the official leaderboard\footnote{\url{https://codalab.lisn.upsaclay.fr/competitions/3770#results}} under the two settings, respectively, ranking the first place in all submissions since December 2023. Our code is available at https://github.com/DeepLearnXMU/CoRE-NEPD.
△ Less
Submitted 24 June, 2024;
originally announced June 2024.
-
Relaxing Continuous Constraints of Equivariant Graph Neural Networks for Physical Dynamics Learning
Authors:
Zinan Zheng,
Yang Liu,
Jia Li,
Jianhua Yao,
Yu Rong
Abstract:
Incorporating Euclidean symmetries (e.g. rotation equivariance) as inductive biases into graph neural networks has improved their generalization ability and data efficiency in unbounded physical dynamics modeling. However, in various scientific and engineering applications, the symmetries of dynamics are frequently discrete due to the boundary conditions. Thus, existing GNNs either overlook necess…
▽ More
Incorporating Euclidean symmetries (e.g. rotation equivariance) as inductive biases into graph neural networks has improved their generalization ability and data efficiency in unbounded physical dynamics modeling. However, in various scientific and engineering applications, the symmetries of dynamics are frequently discrete due to the boundary conditions. Thus, existing GNNs either overlook necessary symmetry, resulting in suboptimal representation ability, or impose excessive equivariance, which fails to generalize to unobserved symmetric dynamics. In this work, we propose a general Discrete Equivariant Graph Neural Network (DEGNN) that guarantees equivariance to a given discrete point group. Specifically, we show that such discrete equivariant message passing could be constructed by transforming geometric features into permutation-invariant embeddings. Through relaxing continuous equivariant constraints, DEGNN can employ more geometric feature combinations to approximate unobserved physical object interaction functions. Two implementation approaches of DEGNN are proposed based on ranking or pooling permutation-invariant functions. We apply DEGNN to various physical dynamics, ranging from particle, molecular, crowd to vehicle dynamics. In twenty scenarios, DEGNN significantly outperforms existing state-of-the-art approaches. Moreover, we show that DEGNN is data efficient, learning with less data, and can generalize across scenarios such as unobserved orientation.
△ Less
Submitted 23 June, 2024;
originally announced June 2024.
-
Optimal Transmit Signal Design for Multi-Target MIMO Sensing Exploiting Prior Information
Authors:
Jiayi Yao,
Shuowen Zhang
Abstract:
In this paper, we study the transmit signal optimization in a multiple-input multiple-output (MIMO) radar system for sensing the angle information of multiple targets via their reflected echo signals. We consider a challenging and practical scenario where the angles to be sensed are unknown and random, while their probability information is known a priori for exploitation. First, we establish an a…
▽ More
In this paper, we study the transmit signal optimization in a multiple-input multiple-output (MIMO) radar system for sensing the angle information of multiple targets via their reflected echo signals. We consider a challenging and practical scenario where the angles to be sensed are unknown and random, while their probability information is known a priori for exploitation. First, we establish an analytical framework to quantify the multi-target sensing performance exploiting prior distribution information, by deriving the posterior Cramér-Rao bound (PCRB) as a lower bound of the mean-squared error (MSE) matrix in sensing multiple unknown and random angles. Then, we formulate and study the transmit sample covariance matrix optimization problem to minimize the PCRB for the sum MSE in estimating all angles. By leveraging Lagrange duality theory, we analytically prove that the optimal transmit covariance matrix has a rank-one structure, despite the multiple angles to be sensed and the continuous feasible range of each angle. Moreover, we propose a sum-of-ratios iterative algorithm which can obtain the optimal solution to the PCRB-minimization problem with low complexity. Numerical results validate our results and the superiority of our proposed design over benchmark schemes.
△ Less
Submitted 21 June, 2024;
originally announced June 2024.
-
ForSE+: Simulating non-Gaussian CMB foregrounds at 3 arcminutes in a stochastic way based on a generative adversarial network
Authors:
Jian Yao,
Nicoletta Krachmalnicoff,
Marianna Foschi,
Giuseppe Puglisi,
Carlo Baccigalupi
Abstract:
We present ForSE+, a Python package that produces non-Gaussian diffuse Galactic thermal dust emission maps at arcminute angular scales and that has the capacity to generate random realizations of small scales. This represents an extension of the ForSE (Foreground Scale Extender) package, which was recently proposed to simulate non-Gaussian small scales of thermal dust emission using generative adv…
▽ More
We present ForSE+, a Python package that produces non-Gaussian diffuse Galactic thermal dust emission maps at arcminute angular scales and that has the capacity to generate random realizations of small scales. This represents an extension of the ForSE (Foreground Scale Extender) package, which was recently proposed to simulate non-Gaussian small scales of thermal dust emission using generative adversarial networks (GANs). With the input of the large-scale polarization maps from observations, ForSE+ has been trained to produce realistic polarized small scales at 3' following the statistical properties, mainly the non-Gaussianity, of observed intensity small scales, which are evaluated through Minkowski functionals. Furthermore, by adding different realizations of random components to the large-scale foregrounds, we show that ForSE+ is able to generate small scales in a stochastic way. In both cases, the output small scales have a similar level of non-Gaussianity compared with real observations and correct amplitude scaling as a power law. These realistic new maps will be useful, in the future, to understand the impact of non-Gaussian foregrounds on the measurements of the cosmic microwave background (CMB) signal, particularly on the lensing reconstruction, de-lensing, and the detection of cosmological gravitational waves in CMB polarization B-modes.
△ Less
Submitted 20 June, 2024;
originally announced June 2024.
-
Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapters
Authors:
Yuhang Zhou,
Zihua Zhao,
Haolin Li,
Siyuan Du,
Jiangchao Yao,
Ya Zhang,
Yanfeng Wang
Abstract:
Training a unified model to take multiple targets into account is a trend towards artificial general intelligence. However, how to efficiently mitigate the training conflicts among heterogeneous data collected from different domains or tasks remains under-explored. In this study, we explore to leverage Mixture of Low-rank Adapters (MoLA) to mitigate conflicts in heterogeneous data training, which…
▽ More
Training a unified model to take multiple targets into account is a trend towards artificial general intelligence. However, how to efficiently mitigate the training conflicts among heterogeneous data collected from different domains or tasks remains under-explored. In this study, we explore to leverage Mixture of Low-rank Adapters (MoLA) to mitigate conflicts in heterogeneous data training, which requires to jointly train the multiple low-rank adapters and their shared backbone. Specifically, we introduce two variants of MoLA, namely, MoLA-Grad and MoLA-Router, to respectively handle the target-aware and target-agnostic scenarios during inference. The former uses task identifiers to assign personalized low-rank adapters to each task, disentangling task-specific knowledge towards their adapters, thereby mitigating heterogeneity conflicts. The latter uses a novel Task-wise Decorrelation (TwD) loss to intervene the router to learn oriented weight combinations of adapters to homogeneous tasks, achieving similar effects. We conduct comprehensive experiments to verify the superiority of MoLA over previous state-of-the-art methods and present in-depth analysis on its working mechanism. Source code is available at: https://github.com/MediaBrain-SJTU/MoLA
△ Less
Submitted 13 June, 2024;
originally announced June 2024.
-
SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models
Authors:
Kehua Feng,
Keyan Ding,
Weijie Wang,
Xiang Zhuang,
Zeyuan Wang,
Ming Qin,
Yu Zhao,
Jianhua Yao,
Qiang Zhang,
Huajun Chen
Abstract:
The burgeoning utilization of Large Language Models (LLMs) in scientific research necessitates advanced benchmarks capable of evaluating their understanding and application of scientific knowledge comprehensively. To address this need, we introduce the SciKnowEval benchmark, a novel framework that systematically evaluates LLMs across five progressive levels of scientific knowledge: studying extens…
▽ More
The burgeoning utilization of Large Language Models (LLMs) in scientific research necessitates advanced benchmarks capable of evaluating their understanding and application of scientific knowledge comprehensively. To address this need, we introduce the SciKnowEval benchmark, a novel framework that systematically evaluates LLMs across five progressive levels of scientific knowledge: studying extensively, inquiring earnestly, thinking profoundly, discerning clearly, and practicing assiduously. These levels aim to assess the breadth and depth of scientific knowledge in LLMs, including knowledge coverage, inquiry and exploration capabilities, reflection and reasoning abilities, ethic and safety considerations, as well as practice proficiency. Specifically, we take biology and chemistry as the two instances of SciKnowEval and construct a dataset encompassing 50K multi-level scientific problems and solutions. By leveraging this dataset, we benchmark 20 leading open-source and proprietary LLMs using zero-shot and few-shot prompting strategies. The results reveal that despite achieving state-of-the-art performance, the proprietary LLMs still have considerable room for improvement, particularly in addressing scientific computations and applications. We anticipate that SciKnowEval will establish a comprehensive standard for benchmarking LLMs in science research and discovery, and promote the development of LLMs that integrate scientific knowledge with strong safety awareness. The dataset and code are publicly available at https://github.com/hicai-zju/sciknoweval .
△ Less
Submitted 13 June, 2024;
originally announced June 2024.
-
Boundary sources of velocity gradient tensor and its invariants
Authors:
Tao Chen,
Jie-Zhi Wu,
Tianshu Liu,
Jie Yao
Abstract:
The present work elucidates the boundary behaviors of the velocity gradient tensor ($\bm{A}\equiv\bm{\nabla}\bm{u}$) and its principal invariants ($P,Q,R$) for compressible flow interacting with a stationary rigid wall. Firstly, it is found that the well-known Caswell formula exhibits an inherent physical structure being compatible with the normal-nilpotent decomposition, where both the strain-rat…
▽ More
The present work elucidates the boundary behaviors of the velocity gradient tensor ($\bm{A}\equiv\bm{\nabla}\bm{u}$) and its principal invariants ($P,Q,R$) for compressible flow interacting with a stationary rigid wall. Firstly, it is found that the well-known Caswell formula exhibits an inherent physical structure being compatible with the normal-nilpotent decomposition, where both the strain-rate and rotation-rate tensors contain the physical effects from the spin component of the vorticity. Secondly, we derive the kinematic and dynamic forms of the boundary $\bm{A}$-flux from which the known boundary fluxes can be recovered by applying the symmetric-antisymmetric decomposition. Then, we obtain the explicit expression of the boundary $Q$ flux as a result of the competition among the boundary fluxes of squared dilatation, enstrophy and squared strain-rate. Importantly, we emphasize that both the coupling between the spin and surface pressure gradient, and the spin-curvature quadratic interaction, are \textit{not} responsible for the generation of the boundary $Q$ flux, although they contribute to both the boundary fluxes of enstrophy and squared strain-rate. Moreover, we prove that the boundary $R$ flux must vanish on a stationary rigid wall. Finally, the boundary fluxes of the invariants of the strain-rate and rotation-rate tensors are also discussed. It is revealed that the boundary flux of the third invariant of the strain-rate tensor is proportional to the wall-normal derivative of the vortex stretching term, which serves as a source term accounting for the the spatiotemporal evolution rate of the wall-normal enstrophy flux. These theoretical results provide a unified description of boundary vorticity and vortex dynamics, which could be valuable in understanding the formation mechanisms of complex near-wall coherent structures and the boundary sources of flow noise.
△ Less
Submitted 24 June, 2024; v1 submitted 12 June, 2024;
originally announced June 2024.
-
Decoupling the Class Label and the Target Concept in Machine Unlearning
Authors:
Jianing Zhu,
Bo Han,
Jiangchao Yao,
Jianliang Xu,
Gang Niu,
Masashi Sugiyama
Abstract:
Machine unlearning as an emerging research topic for data regulations, aims to adjust a trained model to approximate a retrained one that excludes a portion of training data. Previous studies showed that class-wise unlearning is successful in forgetting the knowledge of a target class, through gradient ascent on the forgetting data or fine-tuning with the remaining data. However, while these metho…
▽ More
Machine unlearning as an emerging research topic for data regulations, aims to adjust a trained model to approximate a retrained one that excludes a portion of training data. Previous studies showed that class-wise unlearning is successful in forgetting the knowledge of a target class, through gradient ascent on the forgetting data or fine-tuning with the remaining data. However, while these methods are useful, they are insufficient as the class label and the target concept are often considered to coincide. In this work, we decouple them by considering the label domain mismatch and investigate three problems beyond the conventional all matched forgetting, e.g., target mismatch, model mismatch, and data mismatch forgetting. We systematically analyze the new challenges in restrictively forgetting the target concept and also reveal crucial forgetting dynamics in the representation level to realize these tasks. Based on that, we propose a general framework, namely, TARget-aware Forgetting (TARF). It enables the additional tasks to actively forget the target concept while maintaining the rest part, by simultaneously conducting annealed gradient ascent on the forgetting data and selected gradient descent on the hard-to-affect remaining data. Empirically, various experiments under the newly introduced settings are conducted to demonstrate the effectiveness of our TARF.
△ Less
Submitted 16 June, 2024; v1 submitted 12 June, 2024;
originally announced June 2024.
-
2nd Place Solution for MOSE Track in CVPR 2024 PVUW workshop: Complex Video Object Segmentation
Authors:
Zhensong Xu,
Jiangtao Yao,
Chengjing Wu,
Ting Liu,
Luoqi Liu
Abstract:
Complex video object segmentation serves as a fundamental task for a wide range of downstream applications such as video editing and automatic data annotation. Here we present the 2nd place solution in the MOSE track of PVUW 2024. To mitigate problems caused by tiny objects, similar objects and fast movements in MOSE. We use instance segmentation to generate extra pretraining data from the valid a…
▽ More
Complex video object segmentation serves as a fundamental task for a wide range of downstream applications such as video editing and automatic data annotation. Here we present the 2nd place solution in the MOSE track of PVUW 2024. To mitigate problems caused by tiny objects, similar objects and fast movements in MOSE. We use instance segmentation to generate extra pretraining data from the valid and test set of MOSE. The segmented instances are combined with objects extracted from COCO to augment the training data and enhance semantic representation of the baseline model. Besides, motion blur is added during training to increase robustness against image blur induced by motion. Finally, we apply test time augmentation (TTA) and memory strategy to the inference stage. Our method ranked 2nd in the MOSE track of PVUW 2024, with a $\mathcal{J}$ of 0.8007, a $\mathcal{F}$ of 0.8683 and a $\mathcal{J}$\&$\mathcal{F}$ of 0.8345.
△ Less
Submitted 12 June, 2024;
originally announced June 2024.
-
DualVC 3: Leveraging Language Model Generated Pseudo Context for End-to-end Low Latency Streaming Voice Conversion
Authors:
Ziqian Ning,
Shuai Wang,
Pengcheng Zhu,
Zhichao Wang,
Jixun Yao,
Lei Xie,
Mengxiao Bi
Abstract:
Streaming voice conversion has become increasingly popular for its potential in real-time applications. The recently proposed DualVC 2 has achieved robust and high-quality streaming voice conversion with a latency of about 180ms. Nonetheless, the recognition-synthesis framework hinders end-to-end optimization, and the instability of automatic speech recognition (ASR) model with short chunks makes…
▽ More
Streaming voice conversion has become increasingly popular for its potential in real-time applications. The recently proposed DualVC 2 has achieved robust and high-quality streaming voice conversion with a latency of about 180ms. Nonetheless, the recognition-synthesis framework hinders end-to-end optimization, and the instability of automatic speech recognition (ASR) model with short chunks makes it challenging to further reduce latency. To address these issues, we propose an end-to-end model, DualVC 3. With speaker-independent semantic tokens to guide the training of the content encoder, the dependency on ASR is removed and the model can operate under extremely small chunks, with cascading errors eliminated. A language model is trained on the content encoder output to produce pseudo context by iteratively predicting future frames, providing more contextual information for the decoder to improve conversion quality. Experimental results demonstrate that DualVC 3 achieves comparable performance to DualVC 2 in subjective and objective metrics, with a latency of only 50 ms.
△ Less
Submitted 11 June, 2024;
originally announced June 2024.
-
The measurement of the splashback radius of dark matter halo
Authors:
Weiwei Xu,
Huanyuan Shan,
Ran Li,
Ji Yao,
Chunxiang Wang,
Nan Li,
Chaoli Zhang
Abstract:
In the hierarchical evolution framework of cosmology, larger halos grow through matter accretion and halo mergers. To clarify the halo evolution, we need to define the halo mass and radius physically. However, the pseudo-evolution problem makes the process difficult. Thus, we aim to measure the splashback radius, a physically defined halo radius for a large number of halos with various mass and re…
▽ More
In the hierarchical evolution framework of cosmology, larger halos grow through matter accretion and halo mergers. To clarify the halo evolution, we need to define the halo mass and radius physically. However, the pseudo-evolution problem makes the process difficult. Thus, we aim to measure the splashback radius, a physically defined halo radius for a large number of halos with various mass and redshift, and to determine the most important parameters to affect it. We use the typical definition of splashback radius (Rsp) as the radius with the steepest radial density profile. In this work, we measure Rsp of dark matter halos within the mass of 1e13-3e15Msun and redshifts spanning 0.08-0.65. This is the measurement of the Rsp in the largest range of halo mass and redshift. Using the shear catalog of the DECaLS DR8, we investigate Rsp of halos associated with galaxies and galaxy clusters identified in the various catalogs. Our finding reveals a trend wherein massive halos demonstrate a larger Rsp, and the normalized splashback radius (Rsp/R200m) shows a U-shaped mass evolution. The upturn in these relations mainly comes from the contribution of massive halos with low redshifts. We further find Rsp increases with the peak height, while Rsp/R200m has a negative relation with the peak height. We also find the Rsp >~R200m for most halos, indicating their low accretion rates. Our result is consistent with previous literature across a wide range of mass, redshift, and peak height, as well as the simulation work from More et al. (2015).
△ Less
Submitted 10 June, 2024;
originally announced June 2024.
-
Diversified Batch Selection for Training Acceleration
Authors:
Feng Hong,
Yueming Lyu,
Jiangchao Yao,
Ya Zhang,
Ivor W. Tsang,
Yanfeng Wang
Abstract:
The remarkable success of modern machine learning models on large datasets often demands extensive training time and resource consumption. To save cost, a prevalent research line, known as online batch selection, explores selecting informative subsets during the training process. Although recent efforts achieve advancements by measuring the impact of each sample on generalization, their reliance o…
▽ More
The remarkable success of modern machine learning models on large datasets often demands extensive training time and resource consumption. To save cost, a prevalent research line, known as online batch selection, explores selecting informative subsets during the training process. Although recent efforts achieve advancements by measuring the impact of each sample on generalization, their reliance on additional reference models inherently limits their practical applications, when there are no such ideal models available. On the other hand, the vanilla reference-model-free methods involve independently scoring and selecting data in a sample-wise manner, which sacrifices the diversity and induces the redundancy. To tackle this dilemma, we propose Diversified Batch Selection (DivBS), which is reference-model-free and can efficiently select diverse and representative samples. Specifically, we define a novel selection objective that measures the group-wise orthogonalized representativeness to combat the redundancy issue of previous sample-wise criteria, and provide a principled selection-efficient realization. Extensive experiments across various tasks demonstrate the significant superiority of DivBS in the performance-speedup trade-off. The code is publicly available.
△ Less
Submitted 7 June, 2024;
originally announced June 2024.
-
What is the Best Way for ChatGPT to Translate Poetry?
Authors:
Shanshan Wang,
Derek F. Wong,
Jingming Yao,
Lidia S. Chao
Abstract:
Machine translation (MT) has historically faced significant challenges when applied to literary works, particularly in the domain of poetry translation. The advent of Large Language Models such as ChatGPT holds potential for innovation in this field. This study examines ChatGPT's capabilities in English-Chinese poetry translation tasks, utilizing targeted prompts and small sample scenarios to asce…
▽ More
Machine translation (MT) has historically faced significant challenges when applied to literary works, particularly in the domain of poetry translation. The advent of Large Language Models such as ChatGPT holds potential for innovation in this field. This study examines ChatGPT's capabilities in English-Chinese poetry translation tasks, utilizing targeted prompts and small sample scenarios to ascertain optimal performance. Despite promising outcomes, our analysis reveals persistent issues in the translations generated by ChatGPT that warrant attention. To address these shortcomings, we propose an Explanation-Assisted Poetry Machine Translation (EAPMT) method, which leverages monolingual poetry explanation as a guiding information for the translation process. Furthermore, we refine existing evaluation criteria to better suit the nuances of modern poetry translation. We engaged a panel of professional poets for assessments, complemented evaluations by using GPT-4. The results from both human and machine evaluations demonstrate that our EAPMT method outperforms traditional translation methods of ChatGPT and the existing online systems. This paper validates the efficacy of our method and contributes a novel perspective to machine-assisted literary translation.
△ Less
Submitted 5 June, 2024;
originally announced June 2024.
-
Deterministic Reversible Data Augmentation for Neural Machine Translation
Authors:
Jiashu Yao,
Heyan Huang,
Zeming Liu,
Yuhang Guo
Abstract:
Data augmentation is an effective way to diversify corpora in machine translation, but previous methods may introduce semantic inconsistency between original and augmented data because of irreversible operations and random subword sampling procedures. To generate both symbolically diverse and semantically consistent augmentation data, we propose Deterministic Reversible Data Augmentation (DRDA), a…
▽ More
Data augmentation is an effective way to diversify corpora in machine translation, but previous methods may introduce semantic inconsistency between original and augmented data because of irreversible operations and random subword sampling procedures. To generate both symbolically diverse and semantically consistent augmentation data, we propose Deterministic Reversible Data Augmentation (DRDA), a simple but effective data augmentation method for neural machine translation. DRDA adopts deterministic segmentations and reversible operations to generate multi-granularity subword representations and pulls them closer together with multi-view techniques. With no extra corpora or model changes required, DRDA outperforms strong baselines on several translation tasks with a clear margin (up to 4.3 BLEU gain over Transformer) and exhibits good robustness in noisy, low-resource, and cross-domain datasets.
△ Less
Submitted 4 June, 2024;
originally announced June 2024.
-
Towards Out-of-Distribution Detection in Vocoder Recognition via Latent Feature Reconstruction
Authors:
Renmingyue Du,
Jixun Yao,
Qiuqiang Kong,
Yin Cao
Abstract:
Advancements in synthesized speech have created a growing threat of impersonation, making it crucial to develop deepfake algorithm recognition. One significant aspect is out-of-distribution (OOD) detection, which has gained notable attention due to its important role in deepfake algorithm recognition. However, most of the current approaches for detecting OOD in deepfake algorithm recognition rely…
▽ More
Advancements in synthesized speech have created a growing threat of impersonation, making it crucial to develop deepfake algorithm recognition. One significant aspect is out-of-distribution (OOD) detection, which has gained notable attention due to its important role in deepfake algorithm recognition. However, most of the current approaches for detecting OOD in deepfake algorithm recognition rely on probability-score or classified-distance, which may lead to limitations in the accuracy of the sample at the edge of the threshold. In this study, we propose a reconstruction-based detection approach that employs an autoencoder architecture to compress and reconstruct the acoustic feature extracted from a pre-trained WavLM model. Each acoustic feature belonging to a specific vocoder class is only aptly reconstructed by its corresponding decoder. When none of the decoders can satisfactorily reconstruct a feature, it is classified as an OOD sample. To enhance the distinctiveness of the reconstructed features by each decoder, we incorporate contrastive learning and an auxiliary classifier to further constrain the reconstructed feature. Experiments demonstrate that our proposed approach surpasses baseline systems by a relative margin of 10\% in the evaluation dataset. Ablation studies further validate the effectiveness of both the contrastive constraint and the auxiliary classifier within our proposed approach.
△ Less
Submitted 4 June, 2024;
originally announced June 2024.
-
Causal Distillation for Alleviating Performance Heterogeneity in Recommender Systems
Authors:
Shengyu Zhang,
Ziqi Jiang,
Jiangchao Yao,
Fuli Feng,
Kun Kuang,
Zhou Zhao,
Shuo Li,
Hongxia Yang,
Tat-Seng Chua,
Fei Wu
Abstract:
Recommendation performance usually exhibits a long-tail distribution over users -- a small portion of head users enjoy much more accurate recommendation services than the others. We reveal two sources of this performance heterogeneity problem: the uneven distribution of historical interactions (a natural source); and the biased training of recommender models (a model source). As addressing this pr…
▽ More
Recommendation performance usually exhibits a long-tail distribution over users -- a small portion of head users enjoy much more accurate recommendation services than the others. We reveal two sources of this performance heterogeneity problem: the uneven distribution of historical interactions (a natural source); and the biased training of recommender models (a model source). As addressing this problem cannot sacrifice the overall performance, a wise choice is to eliminate the model bias while maintaining the natural heterogeneity. The key to debiased training lies in eliminating the effect of confounders that influence both the user's historical behaviors and the next behavior. The emerging causal recommendation methods achieve this by modeling the causal effect between user behaviors, however potentially neglect unobserved confounders (\eg, friend suggestions) that are hard to measure in practice. To address unobserved confounders, we resort to the front-door adjustment (FDA) in causal theory and propose a causal multi-teacher distillation framework (CausalD). FDA requires proper mediators in order to estimate the causal effects of historical behaviors on the next behavior. To achieve this, we equip CausalD with multiple heterogeneous recommendation models to model the mediator distribution. Then, the causal effect estimated by FDA is the expectation of recommendation prediction over the mediator distribution and the prior distribution of historical behaviors, which is technically achieved by multi-teacher ensemble. To pursue efficient inference, CausalD further distills multiple teachers into one student model to directly infer the causal effect for making recommendations.
△ Less
Submitted 31 May, 2024;
originally announced May 2024.
-
Unraveling the Impact of Heterophilic Structures on Graph Positive-Unlabeled Learning
Authors:
Yuhao Wu,
Jiangchao Yao,
Bo Han,
Lina Yao,
Tongliang Liu
Abstract:
While Positive-Unlabeled (PU) learning is vital in many real-world scenarios, its application to graph data still remains under-explored. We unveil that a critical challenge for PU learning on graph lies on the edge heterophily, which directly violates the irreducibility assumption for Class-Prior Estimation (class prior is essential for building PU learning algorithms) and degenerates the latent…
▽ More
While Positive-Unlabeled (PU) learning is vital in many real-world scenarios, its application to graph data still remains under-explored. We unveil that a critical challenge for PU learning on graph lies on the edge heterophily, which directly violates the irreducibility assumption for Class-Prior Estimation (class prior is essential for building PU learning algorithms) and degenerates the latent label inference on unlabeled nodes during classifier training. In response to this challenge, we introduce a new method, named Graph PU Learning with Label Propagation Loss (GPL). Specifically, GPL considers learning from PU nodes along with an intermediate heterophily reduction, which helps mitigate the negative impact of the heterophilic structure. We formulate this procedure as a bilevel optimization that reduces heterophily in the inner loop and efficiently learns a classifier in the outer loop. Extensive experiments across a variety of datasets have shown that GPL significantly outperforms baseline methods, confirming its effectiveness and superiority.
△ Less
Submitted 1 June, 2024; v1 submitted 30 May, 2024;
originally announced May 2024.
-
Federated Learning under Partially Class-Disjoint Data via Manifold Reshaping
Authors:
Ziqing Fan,
Jiangchao Yao,
Ruipeng Zhang,
Lingjuan Lyu,
Ya Zhang,
Yanfeng Wang
Abstract:
Statistical heterogeneity severely limits the performance of federated learning (FL), motivating several explorations e.g., FedProx, MOON and FedDyn, to alleviate this problem. Despite effectiveness, their considered scenario generally requires samples from almost all classes during the local training of each client, although some covariate shifts may exist among clients. In fact, the natural case…
▽ More
Statistical heterogeneity severely limits the performance of federated learning (FL), motivating several explorations e.g., FedProx, MOON and FedDyn, to alleviate this problem. Despite effectiveness, their considered scenario generally requires samples from almost all classes during the local training of each client, although some covariate shifts may exist among clients. In fact, the natural case of partially class-disjoint data (PCDD), where each client contributes a few classes (instead of all classes) of samples, is practical yet underexplored. Specifically, the unique collapse and invasion characteristics of PCDD can induce the biased optimization direction in local training, which prevents the efficiency of federated learning. To address this dilemma, we propose a manifold reshaping approach called FedMR to calibrate the feature space of local training. Our FedMR adds two interplaying losses to the vanilla federated learning: one is intra-class loss to decorrelate feature dimensions for anti-collapse; and the other one is inter-class loss to guarantee the proper margin among categories in the feature expansion. We conduct extensive experiments on a range of datasets to demonstrate that our FedMR achieves much higher accuracy and better communication efficiency. Source code is available at: https://github.com/MediaBrain-SJTU/FedMR.git.
△ Less
Submitted 3 June, 2024; v1 submitted 29 May, 2024;
originally announced May 2024.
-
Federated Learning with Bilateral Curation for Partially Class-Disjoint Data
Authors:
Ziqing Fan,
Ruipeng Zhang,
Jiangchao Yao,
Bo Han,
Ya Zhang,
Yanfeng Wang
Abstract:
Partially class-disjoint data (PCDD), a common yet under-explored data formation where each client contributes a part of classes (instead of all classes) of samples, severely challenges the performance of federated algorithms. Without full classes, the local objective will contradict the global objective, yielding the angle collapse problem for locally missing classes and the space waste problem f…
▽ More
Partially class-disjoint data (PCDD), a common yet under-explored data formation where each client contributes a part of classes (instead of all classes) of samples, severely challenges the performance of federated algorithms. Without full classes, the local objective will contradict the global objective, yielding the angle collapse problem for locally missing classes and the space waste problem for locally existing classes. As far as we know, none of the existing methods can intrinsically mitigate PCDD challenges to achieve holistic improvement in the bilateral views (both global view and local view) of federated learning. To address this dilemma, we are inspired by the strong generalization of simplex Equiangular Tight Frame~(ETF) on the imbalanced data, and propose a novel approach called FedGELA where the classifier is globally fixed as a simplex ETF while locally adapted to the personal distributions. Globally, FedGELA provides fair and equal discrimination for all classes and avoids inaccurate updates of the classifier, while locally it utilizes the space of locally missing classes for locally existing classes. We conduct extensive experiments on a range of datasets to demonstrate that our FedGELA achieves promising performance~(averaged improvement of 3.9% to FedAvg and 1.5% to best baselines) and provide both local and global convergence guarantees. Source code is available at:https://github.com/MediaBrain-SJTU/FedGELA.git.
△ Less
Submitted 29 May, 2024;
originally announced May 2024.
-
Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimization
Authors:
Ziqing Fan,
Shengchao Hu,
Jiangchao Yao,
Gang Niu,
Ya Zhang,
Masashi Sugiyama,
Yanfeng Wang
Abstract:
In federated learning (FL), the multi-step update and data heterogeneity among clients often lead to a loss landscape with sharper minima, degenerating the performance of the resulted global model. Prevalent federated approaches incorporate sharpness-aware minimization (SAM) into local training to mitigate this problem. However, the local loss landscapes may not accurately reflect the flatness of…
▽ More
In federated learning (FL), the multi-step update and data heterogeneity among clients often lead to a loss landscape with sharper minima, degenerating the performance of the resulted global model. Prevalent federated approaches incorporate sharpness-aware minimization (SAM) into local training to mitigate this problem. However, the local loss landscapes may not accurately reflect the flatness of global loss landscape in heterogeneous environments; as a result, minimizing local sharpness and calculating perturbations on client data might not align the efficacy of SAM in FL with centralized training. To overcome this challenge, we propose FedLESAM, a novel algorithm that locally estimates the direction of global perturbation on client side as the difference between global models received in the previous active and current rounds. Besides the improved quality, FedLESAM also speed up federated SAM-based approaches since it only performs once backpropagation in each iteration. Theoretically, we prove a slightly tighter bound than its original FedSAM by ensuring consistent perturbation. Empirically, we conduct comprehensive experiments on four federated benchmark datasets under three partition strategies to demonstrate the superior performance and efficiency of FedLESAM.
△ Less
Submitted 29 May, 2024;
originally announced May 2024.
-
Domain-Inspired Sharpness-Aware Minimization Under Domain Shifts
Authors:
Ruipeng Zhang,
Ziqing Fan,
Jiangchao Yao,
Ya Zhang,
Yanfeng Wang
Abstract:
This paper presents a Domain-Inspired Sharpness-Aware Minimization (DISAM) algorithm for optimization under domain shifts. It is motivated by the inconsistent convergence degree of SAM across different domains, which induces optimization bias towards certain domains and thus impairs the overall convergence. To address this issue, we consider the domain-level convergence consistency in the sharpnes…
▽ More
This paper presents a Domain-Inspired Sharpness-Aware Minimization (DISAM) algorithm for optimization under domain shifts. It is motivated by the inconsistent convergence degree of SAM across different domains, which induces optimization bias towards certain domains and thus impairs the overall convergence. To address this issue, we consider the domain-level convergence consistency in the sharpness estimation to prevent the overwhelming (deficient) perturbations for less (well) optimized domains. Specifically, DISAM introduces the constraint of minimizing variance in the domain loss, which allows the elastic gradient calibration in perturbation generation: when one domain is optimized above the averaging level \textit{w.r.t.} loss, the gradient perturbation towards that domain will be weakened automatically, and vice versa. Under this mechanism, we theoretically show that DISAM can achieve faster overall convergence and improved generalization in principle when inconsistent convergence emerges. Extensive experiments on various domain generalization benchmarks show the superiority of DISAM over a range of state-of-the-art methods. Furthermore, we show the superior efficiency of DISAM in parameter-efficient fine-tuning combined with the pretraining models. The source code is released at https://github.com/MediaBrain-SJTU/DISAM.
△ Less
Submitted 29 May, 2024;
originally announced May 2024.
-
Wireless Federated Learning over Resource-Constrained Networks: Digital versus Analog Transmissions
Authors:
Jiacheng Yao,
Wei Xu,
Zhaohui Yang,
Xiaohu You,
Mehdi Bennis,
H. Vincent Poor
Abstract:
To enable wireless federated learning (FL) in communication resource-constrained networks, two communication schemes, i.e., digital and analog ones, are effective solutions. In this paper, we quantitatively compare these two techniques, highlighting their essential differences as well as respectively suitable scenarios. We first examine both digital and analog transmission schemes, together with a…
▽ More
To enable wireless federated learning (FL) in communication resource-constrained networks, two communication schemes, i.e., digital and analog ones, are effective solutions. In this paper, we quantitatively compare these two techniques, highlighting their essential differences as well as respectively suitable scenarios. We first examine both digital and analog transmission schemes, together with a unified and fair comparison framework under imbalanced device sampling, strict latency targets, and transmit power constraints. A universal convergence analysis under various imperfections is established for evaluating the performance of FL over wireless networks. These analytical results reveal that the fundamental difference between the digital and analog communications lies in whether communication and computation are jointly designed or not. The digital scheme decouples the communication design from FL computing tasks, making it difficult to support uplink transmission from massive devices with limited bandwidth and hence the performance is mainly communication-limited. In contrast, the analog communication allows over-the-air computation (AirComp) and achieves better spectrum utilization. However, the computation-oriented analog transmission reduces power efficiency, and its performance is sensitive to computation errors from imperfect channel state information (CSI). Furthermore, device sampling for both schemes are optimized and differences in sampling optimization are analyzed. Numerical results verify the theoretical analysis and affirm the superior performance of the sampling optimization.
△ Less
Submitted 27 May, 2024;
originally announced May 2024.
-
Thermalization and Criticality on an Analog-Digital Quantum Simulator
Authors:
Trond I. Andersen,
Nikita Astrakhantsev,
Amir H. Karamlou,
Julia Berndtsson,
Johannes Motruk,
Aaron Szasz,
Jonathan A. Gross,
Alexander Schuckert,
Tom Westerhout,
Yaxing Zhang,
Ebrahim Forati,
Dario Rossi,
Bryce Kobrin,
Agustin Di Paolo,
Andrey R. Klots,
Ilya Drozdov,
Vladislav D. Kurilovich,
Andre Petukhov,
Lev B. Ioffe,
Andreas Elben,
Aniket Rath,
Vittorio Vitale,
Benoit Vermersch,
Rajeev Acharya,
Laleh Aghababaie Beni
, et al. (202 additional authors not shown)
Abstract:
Understanding how interacting particles approach thermal equilibrium is a major challenge of quantum simulators. Unlocking the full potential of such systems toward this goal requires flexible initial state preparation, precise time evolution, and extensive probes for final state characterization. We present a quantum simulator comprising 69 superconducting qubits which supports both universal qua…
▽ More
Understanding how interacting particles approach thermal equilibrium is a major challenge of quantum simulators. Unlocking the full potential of such systems toward this goal requires flexible initial state preparation, precise time evolution, and extensive probes for final state characterization. We present a quantum simulator comprising 69 superconducting qubits which supports both universal quantum gates and high-fidelity analog evolution, with performance beyond the reach of classical simulation in cross-entropy benchmarking experiments. Emulating a two-dimensional (2D) XY quantum magnet, we leverage a wide range of measurement techniques to study quantum states after ramps from an antiferromagnetic initial state. We observe signatures of the classical Kosterlitz-Thouless phase transition, as well as strong deviations from Kibble-Zurek scaling predictions attributed to the interplay between quantum and classical coarsening of the correlated domains. This interpretation is corroborated by injecting variable energy density into the initial state, which enables studying the effects of the eigenstate thermalization hypothesis (ETH) in targeted parts of the eigenspectrum. Finally, we digitally prepare the system in pairwise-entangled dimer states and image the transport of energy and vorticity during thermalization. These results establish the efficacy of superconducting analog-digital quantum processors for preparing states across many-body spectra and unveiling their thermalization dynamics.
△ Less
Submitted 8 July, 2024; v1 submitted 27 May, 2024;
originally announced May 2024.
-
Mitigating Noisy Correspondence by Geometrical Structure Consistency Learning
Authors:
Zihua Zhao,
Mengxi Chen,
Tianjie Dai,
Jiangchao Yao,
Bo han,
Ya Zhang,
Yanfeng Wang
Abstract:
Noisy correspondence that refers to mismatches in cross-modal data pairs, is prevalent on human-annotated or web-crawled datasets. Prior approaches to leverage such data mainly consider the application of uni-modal noisy label learning without amending the impact on both cross-modal and intra-modal geometrical structures in multimodal learning. Actually, we find that both structures are effective…
▽ More
Noisy correspondence that refers to mismatches in cross-modal data pairs, is prevalent on human-annotated or web-crawled datasets. Prior approaches to leverage such data mainly consider the application of uni-modal noisy label learning without amending the impact on both cross-modal and intra-modal geometrical structures in multimodal learning. Actually, we find that both structures are effective to discriminate noisy correspondence through structural differences when being well-established. Inspired by this observation, we introduce a Geometrical Structure Consistency (GSC) method to infer the true correspondence. Specifically, GSC ensures the preservation of geometrical structures within and between modalities, allowing for the accurate discrimination of noisy samples based on structural differences. Utilizing these inferred true correspondence labels, GSC refines the learning of geometrical structures by filtering out the noisy samples. Experiments across four cross-modal datasets confirm that GSC effectively identifies noisy samples and significantly outperforms the current leading methods.
△ Less
Submitted 27 May, 2024;
originally announced May 2024.
-
CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion
Authors:
Jiayi Yao,
Hanchen Li,
Yuhan Liu,
Siddhant Ray,
Yihua Cheng,
Qizheng Zhang,
Kuntai Du,
Shan Lu,
Junchen Jiang
Abstract:
Large language models (LLMs) often incorporate multiple text chunks in their inputs to provide the necessary contexts. To speed up the prefill of the long LLM inputs, one can pre-compute the KV cache of a text and re-use the KV cache when the context is reused as the prefix of another LLM input. However, the reused text chunks are not always the input prefix, and when they are not, their precomput…
▽ More
Large language models (LLMs) often incorporate multiple text chunks in their inputs to provide the necessary contexts. To speed up the prefill of the long LLM inputs, one can pre-compute the KV cache of a text and re-use the KV cache when the context is reused as the prefix of another LLM input. However, the reused text chunks are not always the input prefix, and when they are not, their precomputed KV caches cannot be directly used since they ignore the text's cross-attention with the preceding text in the LLM input. Thus, the benefits of reusing KV caches remain largely unrealized.
This paper tackles just one question: when an LLM input contains multiple text chunks, how to quickly combine their precomputed KV caches in order to achieve the same generation quality as the expensive full prefill (i.e., without reusing KV cache)? We present CacheBlend, a scheme that reuses the pre-computed KV caches, regardless prefix or not, and selectively recomputes the KV values of a small subset of tokens to partially update each reused KV cache. In the meantime,the small extra delay for recomputing some tokens can be pipelined with the retrieval of KV caches within the same job,allowing CacheBlend to store KV caches in slower devices with more storage capacity while retrieving them without increasing the inference delay. By comparing CacheBlend with the state-of-the-art KV cache reusing schemes on three open-source LLMs of various sizes and four popular benchmark datasets of different tasks, we show that CacheBlend reduces time-to-first-token (TTFT) by 2.2-3.3X and increases the inference throughput by 2.8-5X, compared with full KV recompute, without compromising generation quality or incurring more storage cost.
△ Less
Submitted 3 June, 2024; v1 submitted 26 May, 2024;
originally announced May 2024.
-
TURNIP: A "Nondeterministic" GPU Runtime with CPU RAM Offload
Authors:
Zhimin Ding,
Jiawen Yao,
Brianna Barrow,
Tania Lorido Botran,
Christopher Jermaine,
Yuxin Tang,
Jiehui Li,
Xinyu Yao,
Sleem Mahmoud Abdelghafar,
Daniel Bourgeois
Abstract:
An obvious way to alleviate memory difficulties in GPU-based AI computing is via CPU offload, where data are moved between GPU and CPU RAM, so inexpensive CPU RAM is used to increase the amount of storage available. While CPU offload is an obvious idea, it can greatly slow down a computation, due to the relatively slow transfer rate between CPU RAM and GPU RAM. Thus, any system for CPU offload nee…
▽ More
An obvious way to alleviate memory difficulties in GPU-based AI computing is via CPU offload, where data are moved between GPU and CPU RAM, so inexpensive CPU RAM is used to increase the amount of storage available. While CPU offload is an obvious idea, it can greatly slow down a computation, due to the relatively slow transfer rate between CPU RAM and GPU RAM. Thus, any system for CPU offload needs to ensure that when such a transfer needs to happen, no computation is blocked waiting for the transfer to finish. One of the key challenges when using CPU offload is that memory transfers introduce nondeterminacy into the system: it is not possible to know before runtime when the transfers will finish, and hence what is the best order of operations to run to ensure there is no blocking. In this paper, we describe TURNIP, which is a system for running AI computations using CPU offload. The key innovation in TURNIP is the compilation of the AI computation into a dependency graph that gives the TURNIP runtime freedom to run operations such as GPU kernel calls in many different orders; at runtime, TURNIP chooses the best order in response to real-time events.
△ Less
Submitted 27 May, 2024; v1 submitted 25 May, 2024;
originally announced May 2024.
-
MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Time
Authors:
Jikun Kang,
Xin Zhe Li,
Xi Chen,
Amirreza Kazemi,
Qianyi Sun,
Boxing Chen,
Dong Li,
Xu He,
Quan He,
Feng Wen,
Jianye Hao,
Jun Yao
Abstract:
Although Large Language Models (LLMs) achieve remarkable performance across various tasks, they often struggle with complex reasoning tasks, such as answering mathematical questions. Recent efforts to address this issue have primarily focused on leveraging mathematical datasets through supervised fine-tuning or self-improvement techniques. However, these methods often depend on high-quality datase…
▽ More
Although Large Language Models (LLMs) achieve remarkable performance across various tasks, they often struggle with complex reasoning tasks, such as answering mathematical questions. Recent efforts to address this issue have primarily focused on leveraging mathematical datasets through supervised fine-tuning or self-improvement techniques. However, these methods often depend on high-quality datasets that are difficult to prepare, or they require substantial computational resources for fine-tuning. Inspired by findings that LLMs know how to produce the right answer but struggle to select the correct reasoning path, we propose a purely inference-based searching method -- MindStar (M*). This method formulates reasoning tasks as searching problems and proposes two search ideas to identify the optimal reasoning paths. We evaluate the M* framework on both the GSM8K and MATH datasets, comparing its performance with existing open and closed-source LLMs. Our results demonstrate that M* significantly enhances the reasoning abilities of open-source models, such as Llama-2-13B and Mistral-7B, and achieves comparable performance to GPT-3.5 and Grok-1, but with substantially reduced model size and computational costs.
△ Less
Submitted 26 June, 2024; v1 submitted 25 May, 2024;
originally announced May 2024.
-
Seamless Integration and Implementation of Distributed Contact and Contactless Vital Sign Monitoring
Authors:
Dingding Liang,
Yang Chen,
Jiawei Gao,
Taixia Shi,
Jianping Yao
Abstract:
Real-time vital sign monitoring is gaining immense significance not only in the medical field but also in personal health management. Facing the needs of different application scenarios of the smart and healthy city in the future, the low-cost, large-scale, scalable, and distributed vital sign monitoring system is of great significance. In this work, a seamlessly integrated contact and contactless…
▽ More
Real-time vital sign monitoring is gaining immense significance not only in the medical field but also in personal health management. Facing the needs of different application scenarios of the smart and healthy city in the future, the low-cost, large-scale, scalable, and distributed vital sign monitoring system is of great significance. In this work, a seamlessly integrated contact and contactless vital sign monitoring system, which can simultaneously implement respiration and heartbeat monitoring, is proposed. In contact vital sign monitoring, the chest wall movement due to respiration and heartbeat is translated into changes in the optical output intensity of a fiber Bragg grating (FBG). The FBG is also an important part of radar signal generation for contactless vital sign monitoring, in which the chest wall movement is translated into phase changes of the radar de-chirped signal. By analyzing the intensity of the FBG output and phase of the radar de-chirped signal, real-time respiration and heartbeat monitoring are realized. In addition, due to the distributed structure of the system and its good integration with the wavelength-division multiplexing optical network, it can be massively scaled by employing more wavelengths. A proof-of-concept experiment is carried out. Contact and contactless respiration and heartbeat monitoring of three people are simultaneously realized. During a monitoring time of 60 s, the maximum absolute measurement errors of respiration and heartbeat rates are 1.6 respirations per minute and 2.3 beats per minute, respectively. The measurement error does not have an obvious change even when the monitoring time is decreased to 5 s.
△ Less
Submitted 24 May, 2024;
originally announced May 2024.