-
Coupling AI and Citizen Science in Creation of Enhanced Training Dataset for Medical Image Segmentation
Authors:
Amir Syahmi,
Xiangrong Lu,
Yinxuan Li,
Haoxuan Yao,
Hanjun Jiang,
Ishita Acharya,
Shiyi Wang,
Yang Nan,
Xiaodan Xing,
Guang Yang
Abstract:
Recent advancements in medical imaging and artificial intelligence (AI) have greatly enhanced diagnostic capabilities, but the development of effective deep learning (DL) models is still constrained by the lack of high-quality annotated datasets. The traditional manual annotation process by medical experts is time- and resource-intensive, limiting the scalability of these datasets. In this work, w…
▽ More
Recent advancements in medical imaging and artificial intelligence (AI) have greatly enhanced diagnostic capabilities, but the development of effective deep learning (DL) models is still constrained by the lack of high-quality annotated datasets. The traditional manual annotation process by medical experts is time- and resource-intensive, limiting the scalability of these datasets. In this work, we introduce a robust and versatile framework that combines AI and crowdsourcing to improve both the quality and quantity of medical image datasets across different modalities. Our approach utilises a user-friendly online platform that enables a diverse group of crowd annotators to label medical images efficiently. By integrating the MedSAM segmentation AI with this platform, we accelerate the annotation process while maintaining expert-level quality through an algorithm that merges crowd-labelled images. Additionally, we employ pix2pixGAN, a generative AI model, to expand the training dataset with synthetic images that capture realistic morphological features. These methods are combined into a cohesive framework designed to produce an enhanced dataset, which can serve as a universal pre-processing pipeline to boost the training of any medical deep learning segmentation model. Our results demonstrate that this framework significantly improves model performance, especially when training data is limited.
△ Less
Submitted 4 September, 2024;
originally announced September 2024.
-
Ultrafast measurement of field-particle energy transfer during chorus emissions in space
Authors:
C. M. Liu,
B. N. Zhao,
J. B. Cao,
C. J. Pollock,
C. T. Russell,
Y. Y. Liu,
X. N. Xing,
P. A. Linqvist,
J. L. Burch
Abstract:
Chorus is one of the strongest electromagnetic emissions naturally occurring in space, and can cause hazardous radiations to humans and satellites1-3. Although chorus has attracted extreme interest and been intensively studied for decades4-7, its generation and evolution remain highly debated, due to the complexity of the underlying physics and the limited capacity of previous spacecraft missions7…
▽ More
Chorus is one of the strongest electromagnetic emissions naturally occurring in space, and can cause hazardous radiations to humans and satellites1-3. Although chorus has attracted extreme interest and been intensively studied for decades4-7, its generation and evolution remain highly debated, due to the complexity of the underlying physics and the limited capacity of previous spacecraft missions7. Chorus has also been believed to be governed by planetary magnetic dipolar fields5,7. Contrary to such conventional expectation, here we report unexpected observations of chorus in the terrestrial neutral sheet where magnetic dipolar effect is absent. Using unprecedentedly high-cadence data from the Magnetospheric Multiscale Mission, we present the first, ultrafast measurements of the wave dispersion relation and electron three-dimensional distributions within the waves, showing smoking-gun evidences for chorus-electron interactions and development of electron holes in the wave phase space. We estimate field-particle energy transfer inside the waves and find that the waves were extracting energy from local thermal electrons, in line with the wave positive growth rate derived from instability analysis. Our observations, opening new pathways for resolving long-standing controversies regarding the chorus emissions, are crucial for understanding nonlinear energy transport ubiquitously observed in space and astrophysical environments.
△ Less
Submitted 23 August, 2024;
originally announced August 2024.
-
A Joint Noise Disentanglement and Adversarial Training Framework for Robust Speaker Verification
Authors:
Xujiang Xing,
Mingxing Xu,
Thomas Fang Zheng
Abstract:
Automatic Speaker Verification (ASV) suffers from performance degradation in noisy conditions. To address this issue, we propose a novel adversarial learning framework that incorporates noise-disentanglement to establish a noise-independent speaker invariant embedding space. Specifically, the disentanglement module includes two encoders for separating speaker related and irrelevant information, re…
▽ More
Automatic Speaker Verification (ASV) suffers from performance degradation in noisy conditions. To address this issue, we propose a novel adversarial learning framework that incorporates noise-disentanglement to establish a noise-independent speaker invariant embedding space. Specifically, the disentanglement module includes two encoders for separating speaker related and irrelevant information, respectively. The reconstruction module serves as a regularization term to constrain the noise. A feature-robust loss is also used to supervise the speaker encoder to learn noise-independent speaker embeddings without losing speaker information. In addition, adversarial training is introduced to discourage the speaker encoder from encoding acoustic condition information for achieving a speaker-invariant embedding space. Experiments on VoxCeleb1 indicate that the proposed method improves the performance of the speaker verification system under both clean and noisy conditions.
△ Less
Submitted 22 August, 2024; v1 submitted 21 August, 2024;
originally announced August 2024.
-
OccMamba: Semantic Occupancy Prediction with State Space Models
Authors:
Heng Li,
Yuenan Hou,
Xiaohan Xing,
Xiao Sun,
Yanyong Zhang
Abstract:
Training deep learning models for semantic occupancy prediction is challenging due to factors such as a large number of occupancy cells, severe occlusion, limited visual cues, complicated driving scenarios, etc. Recent methods often adopt transformer-based architectures given their strong capability in learning input-conditioned weights and long-range relationships. However, transformer-based netw…
▽ More
Training deep learning models for semantic occupancy prediction is challenging due to factors such as a large number of occupancy cells, severe occlusion, limited visual cues, complicated driving scenarios, etc. Recent methods often adopt transformer-based architectures given their strong capability in learning input-conditioned weights and long-range relationships. However, transformer-based networks are notorious for their quadratic computation complexity, seriously undermining their efficacy and deployment in semantic occupancy prediction. Inspired by the global modeling and linear computation complexity of the Mamba architecture, we present the first Mamba-based network for semantic occupancy prediction, termed OccMamba. However, directly applying the Mamba architecture to the occupancy prediction task yields unsatisfactory performance due to the inherent domain gap between the linguistic and 3D domains. To relieve this problem, we present a simple yet effective 3D-to-1D reordering operation, i.e., height-prioritized 2D Hilbert expansion. It can maximally retain the spatial structure of point clouds as well as facilitate the processing of Mamba blocks. Our OccMamba achieves state-of-the-art performance on three prevalent occupancy prediction benchmarks, including OpenOccupancy, SemanticKITTI and SemanticPOSS. Notably, on OpenOccupancy, our OccMamba outperforms the previous state-of-the-art Co-Occ by 3.1% IoU and 3.2% mIoU, respectively. Codes will be released upon publication.
△ Less
Submitted 19 August, 2024;
originally announced August 2024.
-
Beyond the Hype: A dispassionate look at vision-language models in medical scenario
Authors:
Yang Nan,
Huichi Zhou,
Xiaodan Xing,
Guang Yang
Abstract:
Recent advancements in Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities across diverse tasks, garnering significant attention in AI communities. However, their performance and reliability in specialized domains such as medicine remain insufficiently assessed. In particular, most assessments over-concentrate in evaluating VLMs based on simple Visual Question Answering…
▽ More
Recent advancements in Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities across diverse tasks, garnering significant attention in AI communities. However, their performance and reliability in specialized domains such as medicine remain insufficiently assessed. In particular, most assessments over-concentrate in evaluating VLMs based on simple Visual Question Answering (VQA) on multi-modality data, while ignoring the in-depth characteristic of LVLMs. In this study, we introduce RadVUQA, a novel Radiological Visual Understanding and Question Answering benchmark, to comprehensively evaluate existing LVLMs. RadVUQA mainly validates LVLMs across five dimensions: 1) Anatomical understanding, assessing the models' ability to visually identify biological structures; 2) Multimodal comprehension, which involves the capability of interpreting linguistic and visual instructions to produce desired outcomes; 3) Quantitative and spatial reasoning, evaluating the models' spatial awareness and proficiency in combining quantitative analysis with visual and linguistic information; 4) Physiological knowledge, measuring the models' capability to comprehend functions and mechanisms of organs and systems; and 5) Robustness, which assesses the models' capabilities against unharmonised and synthetic data. The results indicate that both generalized LVLMs and medical-specific LVLMs have critical deficiencies with weak multimodal comprehension and quantitative reasoning capabilities. Our findings reveal the large gap between existing LVLMs and clinicians, highlighting the urgent need for more robust and intelligent LVLMs. The code and dataset will be available after the acceptance of this paper.
△ Less
Submitted 16 August, 2024;
originally announced August 2024.
-
Focus on Focus: Focus-oriented Representation Learning and Multi-view Cross-modal Alignment for Glioma Grading
Authors:
Li Pan,
Yupei Zhang,
Qiushi Yang,
Tan Li,
Xiaohan Xing,
Maximus C. F. Yeung,
Zhen Chen
Abstract:
Recently, multimodal deep learning, which integrates histopathology slides and molecular biomarkers, has achieved a promising performance in glioma grading. Despite great progress, due to the intra-modality complexity and inter-modality heterogeneity, existing studies suffer from inadequate histopathology representation learning and inefficient molecular-pathology knowledge alignment. These two is…
▽ More
Recently, multimodal deep learning, which integrates histopathology slides and molecular biomarkers, has achieved a promising performance in glioma grading. Despite great progress, due to the intra-modality complexity and inter-modality heterogeneity, existing studies suffer from inadequate histopathology representation learning and inefficient molecular-pathology knowledge alignment. These two issues hinder existing methods to precisely interpret diagnostic molecular-pathology features, thereby limiting their grading performance. Moreover, the real-world applicability of existing multimodal approaches is significantly restricted as molecular biomarkers are not always available during clinical deployment. To address these problems, we introduce a novel Focus on Focus (FoF) framework with paired pathology-genomic training and applicable pathology-only inference, enhancing molecular-pathology representation effectively. Specifically, we propose a Focus-oriented Representation Learning (FRL) module to encourage the model to identify regions positively or negatively related to glioma grading and guide it to focus on the diagnostic areas with a consistency constraint. To effectively link the molecular biomarkers to morphological features, we propose a Multi-view Cross-modal Alignment (MCA) module that projects histopathology representations into molecular subspaces, aligning morphological features with corresponding molecular biomarker status by supervised contrastive learning. Experiments on the TCGA GBM-LGG dataset demonstrate that our FoF framework significantly improves the glioma grading. Remarkably, our FoF achieves superior performance using only histopathology slides compared to existing multimodal methods. The source code is available at https://github.com/peterlipan/FoF.
△ Less
Submitted 16 August, 2024;
originally announced August 2024.
-
A dual-task mutual learning framework for predicting post-thrombectomy cerebral hemorrhage
Authors:
Caiwen Jiang,
Tianyu Wang,
Xiaodan Xing,
Mianxin Liu,
Guang Yang,
Zhongxiang Ding,
Dinggang Shen
Abstract:
Ischemic stroke is a severe condition caused by the blockage of brain blood vessels, and can lead to the death of brain tissue due to oxygen deprivation. Thrombectomy has become a common treatment choice for ischemic stroke due to its immediate effectiveness. But, it carries the risk of postoperative cerebral hemorrhage. Clinically, multiple CT scans within 0-72 hours post-surgery are used to moni…
▽ More
Ischemic stroke is a severe condition caused by the blockage of brain blood vessels, and can lead to the death of brain tissue due to oxygen deprivation. Thrombectomy has become a common treatment choice for ischemic stroke due to its immediate effectiveness. But, it carries the risk of postoperative cerebral hemorrhage. Clinically, multiple CT scans within 0-72 hours post-surgery are used to monitor for hemorrhage. However, this approach exposes radiation dose to patients, and may delay the detection of cerebral hemorrhage. To address this dilemma, we propose a novel prediction framework for measuring postoperative cerebral hemorrhage using only the patient's initial CT scan. Specifically, we introduce a dual-task mutual learning framework to takes the initial CT scan as input and simultaneously estimates both the follow-up CT scan and prognostic label to predict the occurrence of postoperative cerebral hemorrhage. Our proposed framework incorporates two attention mechanisms, i.e., self-attention and interactive attention. Specifically, the self-attention mechanism allows the model to focus more on high-density areas in the image, which are critical for diagnosis (i.e., potential hemorrhage areas). The interactive attention mechanism further models the dependencies between the interrelated generation and classification tasks, enabling both tasks to perform better than the case when conducted individually. Validated on clinical data, our method can generate follow-up CT scans better than state-of-the-art methods, and achieves an accuracy of 86.37% in predicting follow-up prognostic labels. Thus, our work thus contributes to the timely screening of post-thrombectomy cerebral hemorrhage, and could significantly reform the clinical process of thrombectomy and other similar operations related to stroke.
△ Less
Submitted 1 August, 2024;
originally announced August 2024.
-
CIResDiff: A Clinically-Informed Residual Diffusion Model for Predicting Idiopathic Pulmonary Fibrosis Progression
Authors:
Caiwen Jiang,
Xiaodan Xing,
Zaixin Ou,
Mianxin Liu,
Walsh Simon,
Guang Yang,
Dinggang Shen
Abstract:
The progression of Idiopathic Pulmonary Fibrosis (IPF) significantly correlates with higher patient mortality rates. Early detection of IPF progression is critical for initiating timely treatment, which can effectively slow down the advancement of the disease. However, the current clinical criteria define disease progression requiring two CT scans with a one-year interval, presenting a dilemma: a…
▽ More
The progression of Idiopathic Pulmonary Fibrosis (IPF) significantly correlates with higher patient mortality rates. Early detection of IPF progression is critical for initiating timely treatment, which can effectively slow down the advancement of the disease. However, the current clinical criteria define disease progression requiring two CT scans with a one-year interval, presenting a dilemma: a disease progression is identified only after the disease has already progressed. To this end, in this paper, we develop a novel diffusion model to accurately predict the progression of IPF by generating patient's follow-up CT scan from the initial CT scan. Specifically, from the clinical prior knowledge, we tailor improvements to the traditional diffusion model and propose a Clinically-Informed Residual Diffusion model, called CIResDiff. The key innovations of CIResDiff include 1) performing the target region pre-registration to align the lung regions of two CT scans at different time points for reducing the generation difficulty, 2) adopting the residual diffusion instead of traditional diffusion to enable the model focus more on differences (i.e., lesions) between the two CT scans rather than the largely identical anatomical content, and 3) designing the clinically-informed process based on CLIP technology to integrate lung function information which is highly relevant to diagnosis into the reverse process for assisting generation. Extensive experiments on clinical data demonstrate that our approach can outperform state-of-the-art methods and effectively predict the progression of IPF.
△ Less
Submitted 5 August, 2024; v1 submitted 1 August, 2024;
originally announced August 2024.
-
Retinex-Diffusion: On Controlling Illumination Conditions in Diffusion Models via Retinex Theory
Authors:
Xiaoyan Xing,
Vincent Tao Hu,
Jan Hendrik Metzen,
Konrad Groh,
Sezer Karaoglu,
Theo Gevers
Abstract:
This paper introduces a novel approach to illumination manipulation in diffusion models, addressing the gap in conditional image generation with a focus on lighting conditions. We conceptualize the diffusion model as a black-box image render and strategically decompose its energy function in alignment with the image formation model. Our method effectively separates and controls illumination-relate…
▽ More
This paper introduces a novel approach to illumination manipulation in diffusion models, addressing the gap in conditional image generation with a focus on lighting conditions. We conceptualize the diffusion model as a black-box image render and strategically decompose its energy function in alignment with the image formation model. Our method effectively separates and controls illumination-related properties during the generative process. It generates images with realistic illumination effects, including cast shadow, soft shadow, and inter-reflections. Remarkably, it achieves this without the necessity for learning intrinsic decomposition, finding directions in latent space, or undergoing additional training with new datasets.
△ Less
Submitted 28 July, 2024;
originally announced July 2024.
-
Artificial Immunofluorescence in a Flash: Rapid Synthetic Imaging from Brightfield Through Residual Diffusion
Authors:
Xiaodan Xing,
Chunling Tang,
Siofra Murdoch,
Giorgos Papanastasiou,
Yunzhe Guo,
Xianglu Xiao,
Jan Cross-Zamirski,
Carola-Bibiane Schönlieb,
Kristina Xiao Liang,
Zhangming Niu,
Evandro Fei Fang,
Yinhai Wang,
Guang Yang
Abstract:
Immunofluorescent (IF) imaging is crucial for visualizing biomarker expressions, cell morphology and assessing the effects of drug treatments on sub-cellular components. IF imaging needs extra staining process and often requiring cell fixation, therefore it may also introduce artefects and alter endogenouous cell morphology. Some IF stains are expensive or not readily available hence hindering exp…
▽ More
Immunofluorescent (IF) imaging is crucial for visualizing biomarker expressions, cell morphology and assessing the effects of drug treatments on sub-cellular components. IF imaging needs extra staining process and often requiring cell fixation, therefore it may also introduce artefects and alter endogenouous cell morphology. Some IF stains are expensive or not readily available hence hindering experiments. Recent diffusion models, which synthesise high-fidelity IF images from easy-to-acquire brightfield (BF) images, offer a promising solution but are hindered by training instability and slow inference times due to the noise diffusion process. This paper presents a novel method for the conditional synthesis of IF images directly from BF images along with cell segmentation masks. Our approach employs a Residual Diffusion process that enhances stability and significantly reduces inference time. We performed a critical evaluation against other image-to-image synthesis models, including UNets, GANs, and advanced diffusion models. Our model demonstrates significant improvements in image quality (p<0.05 in MSE, PSNR, and SSIM), inference speed (26 times faster than competing diffusion models), and accurate segmentation results for both nuclei and cell bodies (0.77 and 0.63 mean IOU for nuclei and cell true positives, respectively). This paper is a substantial advancement in the field, providing robust and efficient tools for cell image analysis.
△ Less
Submitted 25 July, 2024;
originally announced July 2024.
-
Mixed-View Panorama Synthesis using Geospatially Guided Diffusion
Authors:
Zhexiao Xiong,
Xin Xing,
Scott Workman,
Subash Khanal,
Nathan Jacobs
Abstract:
We introduce the task of mixed-view panorama synthesis, where the goal is to synthesize a novel panorama given a small set of input panoramas and a satellite image of the area. This contrasts with previous work which only uses input panoramas (same-view synthesis), or an input satellite image (cross-view synthesis). We argue that the mixed-view setting is the most natural to support panorama synth…
▽ More
We introduce the task of mixed-view panorama synthesis, where the goal is to synthesize a novel panorama given a small set of input panoramas and a satellite image of the area. This contrasts with previous work which only uses input panoramas (same-view synthesis), or an input satellite image (cross-view synthesis). We argue that the mixed-view setting is the most natural to support panorama synthesis for arbitrary locations worldwide. A critical challenge is that the spatial coverage of panoramas is uneven, with few panoramas available in many regions of the world. We introduce an approach that utilizes diffusion-based modeling and an attention-based architecture for extracting information from all available input imagery. Experimental results demonstrate the effectiveness of our proposed method. In particular, our model can handle scenarios when the available panoramas are sparse or far from the location of the panorama we are attempting to synthesize.
△ Less
Submitted 12 July, 2024;
originally announced July 2024.
-
Can Generative AI Replace Immunofluorescent Staining Processes? A Comparison Study of Synthetically Generated CellPainting Images from Brightfield
Authors:
Xiaodan Xing,
Siofra Murdoch,
Chunling Tang,
Giorgos Papanastasiou,
Jan Cross-Zamirski,
Yunzhe Guo,
Xianglu Xiao,
Carola-Bibiane Schönlieb,
Yinhai Wang,
Guang Yang
Abstract:
Cell imaging assays utilizing fluorescence stains are essential for observing sub-cellular organelles and their responses to perturbations. Immunofluorescent staining process is routinely in labs, however the recent innovations in generative AI is challenging the idea of IF staining are required. This is especially true when the availability and cost of specific fluorescence dyes is a problem to s…
▽ More
Cell imaging assays utilizing fluorescence stains are essential for observing sub-cellular organelles and their responses to perturbations. Immunofluorescent staining process is routinely in labs, however the recent innovations in generative AI is challenging the idea of IF staining are required. This is especially true when the availability and cost of specific fluorescence dyes is a problem to some labs. Furthermore, staining process takes time and leads to inter-intra technician and hinders downstream image and data analysis, and the reusability of image data for other projects. Recent studies showed the use of generated synthetic immunofluorescence (IF) images from brightfield (BF) images using generative AI algorithms in the literature. Therefore, in this study, we benchmark and compare five models from three types of IF generation backbones, CNN, GAN, and diffusion models, using a publicly available dataset. This paper not only serves as a comparative study to determine the best-performing model but also proposes a comprehensive analysis pipeline for evaluating the efficacy of generators in IF image synthesis. We highlighted the potential of deep learning-based generators for IF image synthesis, while also discussed potential issues and future research directions. Although generative AI shows promise in simplifying cell phenotyping using only BF images with IF staining, further research and validations are needed to address the key challenges of model generalisability, batch effects, feature relevance and computational costs.
△ Less
Submitted 16 July, 2024; v1 submitted 15 June, 2024;
originally announced July 2024.
-
VideoCoT: A Video Chain-of-Thought Dataset with Active Annotation Tool
Authors:
Yan Wang,
Yawen Zeng,
Jingsheng Zheng,
Xiaofen Xing,
Jin Xu,
Xiangmin Xu
Abstract:
Multimodal large language models (MLLMs) are flourishing, but mainly focus on images with less attention than videos, especially in sub-fields such as prompt engineering, video chain-of-thought (CoT), and instruction tuning on videos. Therefore, we try to explore the collection of CoT datasets in videos to lead to video OpenQA and improve the reasoning ability of MLLMs. Unfortunately, making such…
▽ More
Multimodal large language models (MLLMs) are flourishing, but mainly focus on images with less attention than videos, especially in sub-fields such as prompt engineering, video chain-of-thought (CoT), and instruction tuning on videos. Therefore, we try to explore the collection of CoT datasets in videos to lead to video OpenQA and improve the reasoning ability of MLLMs. Unfortunately, making such video CoT datasets is not an easy task. Given that human annotation is too cumbersome and expensive, while machine-generated is not reliable due to the hallucination issue, we develop an automatic annotation tool that combines machine and human experts, under the active learning paradigm. Active learning is an interactive strategy between the model and human experts, in this way, the workload of human labeling can be reduced and the quality of the dataset can be guaranteed. With the help of the automatic annotation tool, we strive to contribute three datasets, namely VideoCoT, TopicQA, TopicCoT. Furthermore, we propose a simple but effective benchmark based on the collected datasets, which exploits CoT to maximize the complex reasoning capabilities of MLLMs. Extensive experiments demonstrate the effectiveness our solution.
△ Less
Submitted 7 July, 2024;
originally announced July 2024.
-
SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking
Authors:
Xingrun Xing,
Boyan Gao,
Zheng Zhang,
David A. Clifton,
Shitao Xiao,
Li Du,
Guoqi Li,
Jiajun Zhang
Abstract:
The recent advancements in large language models (LLMs) with billions of parameters have significantly boosted their performance across various real-world applications. However, the inference processes for these models require substantial energy and computational resources, presenting considerable deployment challenges. In contrast, human brains, which contain approximately 86 billion biological n…
▽ More
The recent advancements in large language models (LLMs) with billions of parameters have significantly boosted their performance across various real-world applications. However, the inference processes for these models require substantial energy and computational resources, presenting considerable deployment challenges. In contrast, human brains, which contain approximately 86 billion biological neurons, exhibit significantly greater energy efficiency compared to LLMs with a similar number of parameters. Inspired by this, we redesign 7 to 70 billion parameter LLMs using bio-plausible spiking mechanisms, emulating the efficient behavior of the human brain. We propose the first spiking large language model as recent LLMs termed SpikeLLM. Coupled with the proposed model, a novel spike-driven quantization framework named Optimal Brain Spiking is introduced to reduce the energy cost and accelerate inference speed via two essential approaches: first (second)-order differentiation-based salient channel detection, and per-channel salient outlier expansion with Generalized Integrate-and-Fire neurons. Our proposed spike-driven quantization can plug in main streams of quantization training methods. In the OmniQuant pipeline, SpikeLLM significantly reduces 25.51% WikiText2 perplexity and improves 3.08% average accuracy of 6 zero-shot datasets on a LLAMA2-7B 4A4W model. In the GPTQ pipeline, SpikeLLM realizes a sparse ternary quantization, which achieves additive in all linear layers. Compared with PB-LLM with similar operations, SpikeLLM also exceeds significantly. We will release our code on GitHub.
△ Less
Submitted 5 July, 2024;
originally announced July 2024.
-
Probing Perfection: The Relentless Art of Meddling for Pulmonary Airway Segmentation from HRCT via a Human-AI Collaboration Based Active Learning Method
Authors:
Shiyi Wang,
Yang Nan,
Sheng Zhang,
Federico Felder,
Xiaodan Xing,
Yingying Fang,
Javier Del Ser,
Simon L F Walsh,
Guang Yang
Abstract:
In pulmonary tracheal segmentation, the scarcity of annotated data is a prevalent issue in medical segmentation. Additionally, Deep Learning (DL) methods face challenges: the opacity of 'black box' models and the need for performance enhancement. Our Human-Computer Interaction (HCI) based models (RS_UNet, LC_UNet, UUNet, and WD_UNet) address these challenges by combining diverse query strategies w…
▽ More
In pulmonary tracheal segmentation, the scarcity of annotated data is a prevalent issue in medical segmentation. Additionally, Deep Learning (DL) methods face challenges: the opacity of 'black box' models and the need for performance enhancement. Our Human-Computer Interaction (HCI) based models (RS_UNet, LC_UNet, UUNet, and WD_UNet) address these challenges by combining diverse query strategies with various DL models. We train four HCI models and repeat these steps: (1) Query Strategy: The HCI models select samples that provide the most additional representative information when labeled in each iteration and identify unlabeled samples with the greatest predictive disparity using Wasserstein Distance, Least Confidence, Entropy Sampling, and Random Sampling. (2) Central line correction: Selected samples are used for expert correction of system-generated tracheal central lines in each training round. (3) Update training dataset: Experts update the training dataset after each DL model's training epoch, enhancing the trustworthiness and performance of the models. (4) Model training: The HCI model is trained using the updated dataset and an enhanced UNet version. Experimental results confirm the effectiveness of these HCI-based approaches, showing that WD-UNet, LC-UNet, UUNet, and RS-UNet achieve comparable or superior performance to state-of-the-art DL models. Notably, WD-UNet achieves this with only 15%-35% of the training data, reducing physician annotation time by 65%-85%.
△ Less
Submitted 23 July, 2024; v1 submitted 3 July, 2024;
originally announced July 2024.
-
Emotion and Intent Joint Understanding in Multimodal Conversation: A Benchmarking Dataset
Authors:
Rui Liu,
Haolin Zuo,
Zheng Lian,
Xiaofen Xing,
Björn W. Schuller,
Haizhou Li
Abstract:
Emotion and Intent Joint Understanding in Multimodal Conversation (MC-EIU) aims to decode the semantic information manifested in a multimodal conversational history, while inferring the emotions and intents simultaneously for the current utterance. MC-EIU is enabling technology for many human-computer interfaces. However, there is a lack of available datasets in terms of annotation, modality, lang…
▽ More
Emotion and Intent Joint Understanding in Multimodal Conversation (MC-EIU) aims to decode the semantic information manifested in a multimodal conversational history, while inferring the emotions and intents simultaneously for the current utterance. MC-EIU is enabling technology for many human-computer interfaces. However, there is a lack of available datasets in terms of annotation, modality, language diversity, and accessibility. In this work, we propose an MC-EIU dataset, which features 7 emotion categories, 9 intent categories, 3 modalities, i.e., textual, acoustic, and visual content, and two languages, i.e., English and Mandarin. Furthermore, it is completely open-source for free access. To our knowledge, MC-EIU is the first comprehensive and rich emotion and intent joint understanding dataset for multimodal conversation. Together with the release of the dataset, we also develop an Emotion and Intent Interaction (EI$^2$) network as a reference system by modeling the deep correlation between emotion and intent in the multimodal conversation. With comparative experiments and ablation studies, we demonstrate the effectiveness of the proposed EI$^2$ method on the MC-EIU dataset. The dataset and codes will be made available at: https://github.com/MC-EIU/MC-EIU.
△ Less
Submitted 4 July, 2024; v1 submitted 2 July, 2024;
originally announced July 2024.
-
Evaluating Knowledge-based Cross-lingual Inconsistency in Large Language Models
Authors:
Xiaolin Xing,
Zhiwei He,
Haoyu Xu,
Xing Wang,
Rui Wang,
Yu Hong
Abstract:
This paper investigates the cross-lingual inconsistencies observed in Large Language Models (LLMs), such as ChatGPT, Llama, and Baichuan, which have shown exceptional performance in various Natural Language Processing (NLP) tasks. Despite their successes, these models often exhibit significant inconsistencies when processing the same concepts across different languages. This study focuses on three…
▽ More
This paper investigates the cross-lingual inconsistencies observed in Large Language Models (LLMs), such as ChatGPT, Llama, and Baichuan, which have shown exceptional performance in various Natural Language Processing (NLP) tasks. Despite their successes, these models often exhibit significant inconsistencies when processing the same concepts across different languages. This study focuses on three primary questions: the existence of cross-lingual inconsistencies in LLMs, the specific aspects in which these inconsistencies manifest, and the correlation between cross-lingual consistency and multilingual capabilities of LLMs.To address these questions, we propose an innovative evaluation method for Cross-lingual Semantic Consistency (xSC) using the LaBSE model. We further introduce metrics for Cross-lingual Accuracy Consistency (xAC) and Cross-lingual Timeliness Consistency (xTC) to comprehensively assess the models' performance regarding semantic, accuracy, and timeliness inconsistencies. By harmonizing these metrics, we provide a holistic measurement of LLMs' cross-lingual consistency. Our findings aim to enhance the understanding and improvement of multilingual capabilities and interpretability in LLMs, contributing to the development of more robust and reliable multilingual language models.
△ Less
Submitted 1 July, 2024;
originally announced July 2024.
-
MMR-Mamba: Multi-Modal MRI Reconstruction with Mamba and Spatial-Frequency Information Fusion
Authors:
Jing Zou,
Lanqing Liu,
Qi Chen,
Shujun Wang,
Zhanli Hu,
Xiaohan Xing,
Jing Qin
Abstract:
Multi-modal MRI offers valuable complementary information for diagnosis and treatment; however, its utility is limited by prolonged scanning times. To accelerate the acquisition process, a practical approach is to reconstruct images of the target modality, which requires longer scanning times, from under-sampled k-space data using the fully-sampled reference modality with shorter scanning times as…
▽ More
Multi-modal MRI offers valuable complementary information for diagnosis and treatment; however, its utility is limited by prolonged scanning times. To accelerate the acquisition process, a practical approach is to reconstruct images of the target modality, which requires longer scanning times, from under-sampled k-space data using the fully-sampled reference modality with shorter scanning times as guidance. The primary challenge of this task is comprehensively and efficiently integrating complementary information from different modalities to achieve high-quality reconstruction. Existing methods struggle with this: 1) convolution-based models fail to capture long-range dependencies; 2) transformer-based models, while excelling in global feature modeling, struggle with quadratic computational complexity. To address this, we propose MMR-Mamba, a novel framework that thoroughly and efficiently integrates multi-modal features for MRI reconstruction, leveraging Mamba's capability to capture long-range dependencies with linear computational complexity while exploiting global properties of the Fourier domain. Specifically, we first design a Target modality-guided Cross Mamba (TCM) module in the spatial domain, which maximally restores the target modality information by selectively incorporating relevant information from the reference modality. Then, we introduce a Selective Frequency Fusion (SFF) module to efficiently integrate global information in the Fourier domain and recover high-frequency signals for the reconstruction of structural details. Furthermore, we devise an Adaptive Spatial-Frequency Fusion (ASFF) module, which mutually enhances the spatial and frequency domains by supplementing less informative channels from one domain with corresponding channels from the other.
△ Less
Submitted 7 July, 2024; v1 submitted 27 June, 2024;
originally announced June 2024.
-
Decoding Decision Reasoning: A Counterfactual-Powered Model for Knowledge Discovery
Authors:
Yingying Fang,
Zihao Jin,
Xiaodan Xing,
Simon Walsh,
Guang Yang
Abstract:
In medical imaging, particularly in early disease detection and prognosis tasks, discerning the rationale behind an AI model's predictions is crucial for evaluating the reliability of its decisions. Conventional explanation methods face challenges in identifying discernible decisive features in medical image classifications, where discriminative features are subtle or not immediately apparent. To…
▽ More
In medical imaging, particularly in early disease detection and prognosis tasks, discerning the rationale behind an AI model's predictions is crucial for evaluating the reliability of its decisions. Conventional explanation methods face challenges in identifying discernible decisive features in medical image classifications, where discriminative features are subtle or not immediately apparent. To bridge this gap, we propose an explainable model that is equipped with both decision reasoning and feature identification capabilities. Our approach not only detects influential image patterns but also uncovers the decisive features that drive the model's final predictions. By implementing our method, we can efficiently identify and visualise class-specific features leveraged by the data-driven model, providing insights into the decision-making processes of deep learning models. We validated our model in the demanding realm of medical prognosis task, demonstrating its efficacy and potential in enhancing the reliability of AI in healthcare and in discovering new knowledge in diseases where prognostic understanding is limited.
△ Less
Submitted 23 May, 2024;
originally announced June 2024.
-
Fuzzy Attention-based Border Rendering Network for Lung Organ Segmentation
Authors:
Sheng Zhang,
Yang Nan,
Yingying Fang,
Shiyi Wang,
Xiaodan Xing,
Zhifan Gao,
Guang Yang
Abstract:
Automatic lung organ segmentation on CT images is crucial for lung disease diagnosis. However, the unlimited voxel values and class imbalance of lung organs can lead to false-negative/positive and leakage issues in advanced methods. Additionally, some slender lung organs are easily lost during the recycled down/up-sample procedure, e.g., bronchioles & arterioles, causing severe discontinuity issue…
▽ More
Automatic lung organ segmentation on CT images is crucial for lung disease diagnosis. However, the unlimited voxel values and class imbalance of lung organs can lead to false-negative/positive and leakage issues in advanced methods. Additionally, some slender lung organs are easily lost during the recycled down/up-sample procedure, e.g., bronchioles & arterioles, causing severe discontinuity issue. Inspired by these, this paper introduces an effective lung organ segmentation method called Fuzzy Attention-based Border Rendering (FABR) network. Since fuzzy logic can handle the uncertainty in feature extraction, hence the fusion of deep networks and fuzzy sets should be a viable solution for better performance. Meanwhile, unlike prior top-tier methods that operate on all regular dense points, our FABR depicts lung organ regions as cube-trees, focusing only on recycle-sampled border vulnerable points, rendering the severely discontinuous, false-negative/positive organ regions with a novel Global-Local Cube-tree Fusion (GLCF) module. All experimental results, on four challenging datasets of airway & artery, demonstrate that our method can achieve the favorable performance significantly.
△ Less
Submitted 1 July, 2024; v1 submitted 23 June, 2024;
originally announced June 2024.
-
Competing excitation quenching and charge exchange in ultracold Li-Ba$^+$ collisions
Authors:
Xiaodong Xing,
Pascal Weckesser,
Fabian Thielemann,
Tibor Jónás,
Romain Vexiau,
Nadia Bouloufa-Maafa,
Eliane Luc-Koenig,
Kirk W. Madison,
Andrea Orbán,
Ting Xie,
Tobias Schaetz,
Olivier Dulieu
Abstract:
Hybrid atom-ion systems are a rich and powerful platform for studying chemical reactions, as they feature both excellent control over the electronic state preparation and readout as well as a versatile tunability over the scattering energy, ranging from the few-partial wave regime to the quantum regime. In this work, we make use of these excellent control knobs, and present a joint experimental an…
▽ More
Hybrid atom-ion systems are a rich and powerful platform for studying chemical reactions, as they feature both excellent control over the electronic state preparation and readout as well as a versatile tunability over the scattering energy, ranging from the few-partial wave regime to the quantum regime. In this work, we make use of these excellent control knobs, and present a joint experimental and theoretical study of the collisions of a single $^{138}$Ba$^+$ ion prepared in the $5d\,^2D_{3/2,5/2}$ metastable states with a ground state $^6$Li gas near quantum degeneracy. We show that in contrast to previously reported atom-ion mixtures, several non-radiative processes, including charge exchange, excitation exchange and quenching, compete with each other due to the inherent complexity of the ion-atom molecular structure. We present a full quantum model based on high-level electronic structure calculations involving spin-orbit couplings. Results are in excellent agreement with observations, highlighting the strong coupling between the internal angular momenta and the mechanical rotation of the colliding pair, which is relevant in any other hybrid system composed of an alkali-metal atom and an alkaline-earth ion.
△ Less
Submitted 23 June, 2024;
originally announced June 2024.
-
MIPI 2024 Challenge on Few-shot RAW Image Denoising: Methods and Results
Authors:
Xin Jin,
Chunle Guo,
Xiaoming Li,
Zongsheng Yue,
Chongyi Li,
Shangchen Zhou,
Ruicheng Feng,
Yuekun Dai,
Peiqing Yang,
Chen Change Loy,
Ruoqi Li,
Chang Liu,
Ziyi Wang,
Yao Du,
Jingjing Yang,
Long Bao,
Heng Sun,
Xiangyu Kong,
Xiaoxia Xing,
Jinlong Wu,
Yuanyang Xue,
Hyunhee Park,
Sejun Song,
Changho Kim,
Jingfan Tan
, et al. (17 additional authors not shown)
Abstract:
The increasing demand for computational photography and imaging on mobile platforms has led to the widespread development and integration of advanced image sensors with novel algorithms in camera systems. However, the scarcity of high-quality data for research and the rare opportunity for in-depth exchange of views from industry and academia constrain the development of mobile intelligent photogra…
▽ More
The increasing demand for computational photography and imaging on mobile platforms has led to the widespread development and integration of advanced image sensors with novel algorithms in camera systems. However, the scarcity of high-quality data for research and the rare opportunity for in-depth exchange of views from industry and academia constrain the development of mobile intelligent photography and imaging (MIPI). Building on the achievements of the previous MIPI Workshops held at ECCV 2022 and CVPR 2023, we introduce our third MIPI challenge including three tracks focusing on novel image sensors and imaging algorithms. In this paper, we summarize and review the Few-shot RAW Image Denoising track on MIPI 2024. In total, 165 participants were successfully registered, and 7 teams submitted results in the final testing phase. The developed solutions in this challenge achieved state-of-the-art erformance on Few-shot RAW Image Denoising. More details of this challenge and the link to the dataset can be found at https://mipichallenge.org/MIPI2024.
△ Less
Submitted 11 June, 2024;
originally announced June 2024.
-
SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms
Authors:
Xingrun Xing,
Zheng Zhang,
Ziyi Ni,
Shitao Xiao,
Yiming Ju,
Siqi Fan,
Yequan Wang,
Jiajun Zhang,
Guoqi Li
Abstract:
Towards energy-efficient artificial intelligence similar to the human brain, the bio-inspired spiking neural networks (SNNs) have advantages of biological plausibility, event-driven sparsity, and binary activation. Recently, large-scale language models exhibit promising generalization capability, making it a valuable issue to explore more general spike-driven models. However, the binary spikes in…
▽ More
Towards energy-efficient artificial intelligence similar to the human brain, the bio-inspired spiking neural networks (SNNs) have advantages of biological plausibility, event-driven sparsity, and binary activation. Recently, large-scale language models exhibit promising generalization capability, making it a valuable issue to explore more general spike-driven models. However, the binary spikes in existing SNNs fail to encode adequate semantic information, placing technological challenges for generalization. This work proposes the first fully spiking mechanism for general language tasks, including both discriminative and generative ones. Different from previous spikes with {0,1} levels, we propose a more general spike formulation with bi-directional, elastic amplitude, and elastic frequency encoding, while still maintaining the addition nature of SNNs. In a single time step, the spike is enhanced by direction and amplitude information; in spike frequency, a strategy to control spike firing rate is well designed. We plug this elastic bi-spiking mechanism in language modeling, named SpikeLM. It is the first time to handle general language tasks with fully spike-driven models, which achieve much higher accuracy than previously possible. SpikeLM also greatly bridges the performance gap between SNNs and ANNs in language modeling. Our code is available at https://github.com/Xingrun-Xing/SpikeLM.
△ Less
Submitted 5 June, 2024;
originally announced June 2024.
-
CAMP: Compiler and Allocator-based Heap Memory Protection
Authors:
Zhenpeng Lin,
Zheng Yu,
Ziyi Guo,
Simone Campanoni,
Peter Dinda,
Xinyu Xing
Abstract:
The heap is a critical and widely used component of many applications. Due to its dynamic nature, combined with the complexity of heap management algorithms, it is also a frequent target for security exploits. To enhance the heap's security, various heap protection techniques have been introduced, but they either introduce significant runtime overhead or have limited protection.
We present CAMP,…
▽ More
The heap is a critical and widely used component of many applications. Due to its dynamic nature, combined with the complexity of heap management algorithms, it is also a frequent target for security exploits. To enhance the heap's security, various heap protection techniques have been introduced, but they either introduce significant runtime overhead or have limited protection.
We present CAMP, a new sanitizer for detecting and capturing heap memory corruption. CAMP leverages a compiler and a customized memory allocator. The compiler adds boundary-checking and escape-tracking instructions to the target program, while the memory allocator tracks memory ranges, coordinates with the instrumentation, and neutralizes dangling pointers. With the novel error detection scheme, CAMP enables various compiler optimization strategies and thus eliminates redundant and unnecessary check instrumentation. This design minimizes runtime overhead without sacrificing security guarantees. Our evaluation and comparison of CAMP with existing tools, using both real-world applications and SPEC CPU benchmarks, show that it provides even better heap corruption detection capability with lower runtime overhead.
△ Less
Submitted 4 June, 2024;
originally announced June 2024.
-
Take a Step Further: Understanding Page Spray in Linux Kernel Exploitation
Authors:
Ziyi Guo,
Dang K Le,
Zhenpeng Lin,
Kyle Zeng,
Ruoyu Wang,
Tiffany Bao,
Yan Shoshitaishvili,
Adam Doupé,
Xinyu Xing
Abstract:
Recently, a novel method known as Page Spray emerges, focusing on page-level exploitation for kernel vulnerabilities. Despite the advantages it offers in terms of exploitability, stability, and compatibility, comprehensive research on Page Spray remains scarce. Questions regarding its root causes, exploitation model, comparative benefits over other exploitation techniques, and possible mitigation…
▽ More
Recently, a novel method known as Page Spray emerges, focusing on page-level exploitation for kernel vulnerabilities. Despite the advantages it offers in terms of exploitability, stability, and compatibility, comprehensive research on Page Spray remains scarce. Questions regarding its root causes, exploitation model, comparative benefits over other exploitation techniques, and possible mitigation strategies have largely remained unanswered. In this paper, we conduct a systematic investigation into Page Spray, providing an in-depth understanding of this exploitation technique. We introduce a comprehensive exploit model termed the \sys model, elucidating its fundamental principles. Additionally, we conduct a thorough analysis of the root causes underlying Page Spray occurrences within the Linux Kernel. We design an analyzer based on the Page Spray analysis model to identify Page Spray callsites. Subsequently, we evaluate the stability, exploitability, and compatibility of Page Spray through meticulously designed experiments. Finally, we propose mitigation principles for addressing Page Spray and introduce our own lightweight mitigation approach. This research aims to assist security researchers and developers in gaining insights into Page Spray, ultimately enhancing our collective understanding of this emerging exploitation technique and making improvements to the community.
△ Less
Submitted 6 June, 2024; v1 submitted 3 June, 2024;
originally announced June 2024.
-
Stochastic Thermodynamics of Micromagnetics with Spin Torque
Authors:
Mingnan Ding,
Jun Wu,
Xiangjun Xing
Abstract:
In this work, we study the stochastic dynamics of micro-magnetics interacting with a spin-current torque. We extend the previously constructed stochastic Landau-Lifshitz equation to the case with spin-current torque, and verify the conditions of detailed balance. Then we construct various thermodynamics quantities such as work and heat, and prove the second law of thermodynamics. Due to the existe…
▽ More
In this work, we study the stochastic dynamics of micro-magnetics interacting with a spin-current torque. We extend the previously constructed stochastic Landau-Lifshitz equation to the case with spin-current torque, and verify the conditions of detailed balance. Then we construct various thermodynamics quantities such as work and heat, and prove the second law of thermodynamics. Due to the existence of spin-torque and the asymmetry of the kinetic matrix, a novel effect of entropy pumping shows up. As a consequence, the system may behave as a heat engine which constantly transforms heat into magnetic work. Finally, we derive a fluctuation theorem for the joint probability density function of the pumped entropy and the total work, and verify it using numerical simulations.
△ Less
Submitted 5 August, 2024; v1 submitted 4 June, 2024;
originally announced June 2024.
-
ShadowBound: Efficient Heap Memory Protection Through Advanced Metadata Management and Customized Compiler Optimization
Authors:
Zheng Yu,
Ganxiang Yang,
Xinyu Xing
Abstract:
In software development, the prevalence of unsafe languages such as C and C++ introduces potential vulnerabilities, especially within the heap, a pivotal component for dynamic memory allocation. Despite its significance, heap management complexities have made heap corruption pervasive, posing severe threats to system security. While prior solutions aiming for temporal and spatial memory safety exh…
▽ More
In software development, the prevalence of unsafe languages such as C and C++ introduces potential vulnerabilities, especially within the heap, a pivotal component for dynamic memory allocation. Despite its significance, heap management complexities have made heap corruption pervasive, posing severe threats to system security. While prior solutions aiming for temporal and spatial memory safety exhibit overheads deemed impractical, we present ShadowBound, a unique heap memory protection design. At its core, ShadowBound is an efficient out-of-bounds defense that can work with various use-after-free defenses (e.g. MarkUs, FFMalloc, PUMM) without compatibility constraints. We harness a shadow memory-based metadata management mechanism to store heap chunk boundaries and apply customized compiler optimizations tailored for boundary checking. We implemented ShadowBound atop the LLVM framework and integrated three state-of-the-art use-after-free defenses. Our evaluations show that ShadowBound provides robust heap protection with minimal time and memory overhead, suggesting its effectiveness and efficiency in safeguarding real-world programs against prevalent heap vulnerabilities.
△ Less
Submitted 4 June, 2024;
originally announced June 2024.
-
Decoupled Alignment for Robust Plug-and-Play Adaptation
Authors:
Haozheng Luo,
Jiahao Yu,
Wenxin Zhang,
Jialong Li,
Jerry Yao-Chieh Hu,
Xinyu Xing,
Han Liu
Abstract:
We introduce a low-resource safety enhancement method for aligning large language models (LLMs) without the need for supervised fine-tuning (SFT) or reinforcement learning from human feedback (RLHF). Our main idea is to exploit knowledge distillation to extract the alignment information from existing well-aligned LLMs and integrate it into unaligned LLMs in a plug-and-play fashion. Methodology, we…
▽ More
We introduce a low-resource safety enhancement method for aligning large language models (LLMs) without the need for supervised fine-tuning (SFT) or reinforcement learning from human feedback (RLHF). Our main idea is to exploit knowledge distillation to extract the alignment information from existing well-aligned LLMs and integrate it into unaligned LLMs in a plug-and-play fashion. Methodology, we employ delta debugging to identify the critical components of knowledge necessary for effective distillation. On the harmful question dataset, our method significantly enhances the average defense success rate by approximately 14.41%, reaching as high as 51.39%, in 17 unaligned pre-trained LLMs, without compromising performance.
△ Less
Submitted 6 June, 2024; v1 submitted 3 June, 2024;
originally announced June 2024.
-
Enhancing Jailbreak Attack Against Large Language Models through Silent Tokens
Authors:
Jiahao Yu,
Haozheng Luo,
Jerry Yao-Chieh Hu,
Wenbo Guo,
Han Liu,
Xinyu Xing
Abstract:
Along with the remarkable successes of Language language models, recent research also started to explore the security threats of LLMs, including jailbreaking attacks. Attackers carefully craft jailbreaking prompts such that a target LLM will respond to the harmful question. Existing jailbreaking attacks require either human experts or leveraging complicated algorithms to craft jailbreaking prompts…
▽ More
Along with the remarkable successes of Language language models, recent research also started to explore the security threats of LLMs, including jailbreaking attacks. Attackers carefully craft jailbreaking prompts such that a target LLM will respond to the harmful question. Existing jailbreaking attacks require either human experts or leveraging complicated algorithms to craft jailbreaking prompts. In this paper, we introduce BOOST, a simple attack that leverages only the eos tokens. We demonstrate that rather than constructing complicated jailbreaking prompts, the attacker can simply append a few eos tokens to the end of a harmful question. It will bypass the safety alignment of LLMs and lead to successful jailbreaking attacks. We further apply BOOST to four representative jailbreak methods and show that the attack success rates of these methods can be significantly enhanced by simply adding eos tokens to the prompt. To understand this simple but novel phenomenon, we conduct empirical analyses. Our analysis reveals that adding eos tokens makes the target LLM believe the input is much less harmful, and eos tokens have low attention values and do not affect LLM's understanding of the harmful questions, leading the model to actually respond to the questions. Our findings uncover how fragile an LLM is against jailbreak attacks, motivating the development of strong safety alignment approaches.
△ Less
Submitted 4 June, 2024; v1 submitted 31 May, 2024;
originally announced May 2024.
-
When AI Eats Itself: On the Caveats of Data Pollution in the Era of Generative AI
Authors:
Xiaodan Xing,
Fadong Shi,
Jiahao Huang,
Yinzhe Wu,
Yang Nan,
Sheng Zhang,
Yingying Fang,
Mike Roberts,
Carola-Bibiane Schönlieb,
Javier Del Ser,
Guang Yang
Abstract:
Generative artificial intelligence (AI) technologies and large models are producing realistic outputs across various domains, such as images, text, speech, and music. Creating these advanced generative models requires significant resources, particularly large and high-quality datasets. To minimize training expenses, many algorithm developers use data created by the models themselves as a cost-effe…
▽ More
Generative artificial intelligence (AI) technologies and large models are producing realistic outputs across various domains, such as images, text, speech, and music. Creating these advanced generative models requires significant resources, particularly large and high-quality datasets. To minimize training expenses, many algorithm developers use data created by the models themselves as a cost-effective training solution. However, not all synthetic data effectively improve model performance, necessitating a strategic balance in the use of real versus synthetic data to optimize outcomes.
Currently, the previously well-controlled integration of real and synthetic data is becoming uncontrollable. The widespread and unregulated dissemination of synthetic data online leads to the contamination of datasets traditionally compiled through web scraping, now mixed with unlabeled synthetic data. This trend portends a future where generative AI systems may increasingly rely blindly on consuming self-generated data, raising concerns about model performance and ethical issues. What will happen if generative AI continuously consumes itself without discernment? What measures can we take to mitigate the potential adverse effects?
There is a significant gap in the scientific literature regarding the impact of synthetic data use in generative AI, particularly in terms of the fusion of multimodal information. To address this research gap, this review investigates the consequences of integrating synthetic data blindly on training generative AI on both image and text modalities and explores strategies to mitigate these effects. The goal is to offer a comprehensive view of synthetic data's role, advocating for a balanced approach to its use and exploring practices that promote the sustainable development of generative AI technologies in the era of large models.
△ Less
Submitted 25 July, 2024; v1 submitted 15 May, 2024;
originally announced May 2024.
-
RICE: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanation
Authors:
Zelei Cheng,
Xian Wu,
Jiahao Yu,
Sabrina Yang,
Gang Wang,
Xinyu Xing
Abstract:
Deep reinforcement learning (DRL) is playing an increasingly important role in real-world applications. However, obtaining an optimally performing DRL agent for complex tasks, especially with sparse rewards, remains a significant challenge. The training of a DRL agent can be often trapped in a bottleneck without further progress. In this paper, we propose RICE, an innovative refining scheme for re…
▽ More
Deep reinforcement learning (DRL) is playing an increasingly important role in real-world applications. However, obtaining an optimally performing DRL agent for complex tasks, especially with sparse rewards, remains a significant challenge. The training of a DRL agent can be often trapped in a bottleneck without further progress. In this paper, we propose RICE, an innovative refining scheme for reinforcement learning that incorporates explanation methods to break through the training bottlenecks. The high-level idea of RICE is to construct a new initial state distribution that combines both the default initial states and critical states identified through explanation methods, thereby encouraging the agent to explore from the mixed initial states. Through careful design, we can theoretically guarantee that our refining scheme has a tighter sub-optimality bound. We evaluate RICE in various popular RL environments and real-world applications. The results demonstrate that RICE significantly outperforms existing refining schemes in enhancing agent performance.
△ Less
Submitted 5 June, 2024; v1 submitted 5 May, 2024;
originally announced May 2024.
-
VectorPainter: A Novel Approach to Stylized Vector Graphics Synthesis with Vectorized Strokes
Authors:
Juncheng Hu,
Ximing Xing,
Zhengqi Zhang,
Jing Zhang,
Qian Yu
Abstract:
We propose a novel method, VectorPainter, for the task of stylized vector graphics synthesis. Given a text prompt and a reference style image, VectorPainter generates a vector graphic that aligns in content with the text prompt and remains faithful in style to the reference image. We recognize that the key to this task lies in fully leveraging the intrinsic properties of vector graphics. Innovativ…
▽ More
We propose a novel method, VectorPainter, for the task of stylized vector graphics synthesis. Given a text prompt and a reference style image, VectorPainter generates a vector graphic that aligns in content with the text prompt and remains faithful in style to the reference image. We recognize that the key to this task lies in fully leveraging the intrinsic properties of vector graphics. Innovatively, we conceptualize the stylization process as the rearrangement of vectorized strokes extracted from the reference image. VectorPainter employs an optimization-based pipeline. It begins by extracting vectorized strokes from the reference image, which are then used to initialize the synthesis process. To ensure fidelity to the reference style, a novel style preservation loss is introduced. Extensive experiments have been conducted to demonstrate that our method is capable of aligning with the text description while remaining faithful to the reference image.
△ Less
Submitted 5 May, 2024;
originally announced May 2024.
-
Stochastic thermodynamics of Brownian motion in a flowing fluid
Authors:
Jun Wu,
Mingnan Ding,
Xiangjun Xing
Abstract:
We study stochastic thermodynamics of over-damped Brownian motion in a flowing fluid. Unlike some previous works, we treat the effects of the flow field as a non-conservational driving force acting on the Brownian particle. This allows us to apply the theoretical formalism developed in a recent work for general non-conservative Langevin dynamics. We define heat and work both at the trajectory leve…
▽ More
We study stochastic thermodynamics of over-damped Brownian motion in a flowing fluid. Unlike some previous works, we treat the effects of the flow field as a non-conservational driving force acting on the Brownian particle. This allows us to apply the theoretical formalism developed in a recent work for general non-conservative Langevin dynamics. We define heat and work both at the trajectory level and at the ensemble level, and prove the second law of thermodynamics explicitly. The entropy production (EP) is decomposed into a housekeeping part and an excess part, both of which are non-negative at the ensemble level. Fluctuation theorems are derived for the housekeeping work, the excess work, and the total work, which are further verified using numerical simulations. A comparison between our theory and an earlier theory by Speck et. al. is also carried out.
△ Less
Submitted 21 April, 2024;
originally announced April 2024.
-
Stochastic Thermodynamics of Micromagnetics
Authors:
Mingnan Ding,
Jun Wu,
Xiangjun Xing
Abstract:
In this work, we study the stochastic thermodynamics of micro-magnetic systems. We first formulate the stochastic dynamics of micro-magnetic systems by incorporating noises into Landau-Lifshitz (LL) equation, which describes the irreversible and deterministic dynamics of magnetic moments. The resulting stochastic Landau-Lifshitz (sLL) equation obeys detailed balance, which guarantees that, with th…
▽ More
In this work, we study the stochastic thermodynamics of micro-magnetic systems. We first formulate the stochastic dynamics of micro-magnetic systems by incorporating noises into Landau-Lifshitz (LL) equation, which describes the irreversible and deterministic dynamics of magnetic moments. The resulting stochastic Landau-Lifshitz (sLL) equation obeys detailed balance, which guarantees that, with the external field fixed, the system converges to thermodynamic equilibrium with vanishing entropy production and with non-vanishing probability current. We then discuss various thermodynamic variables both at the trajectory level and at the ensemble level, and further establish both the first and the second laws of thermodynamics. Finally, we establish fluctuation theorems, and verify them using numerical simulations.
△ Less
Submitted 4 August, 2024; v1 submitted 21 April, 2024;
originally announced April 2024.
-
Picturing the Gap Between the Performance and US-DOE's Hydrogen Storage Target: A Data-Driven Model for MgH2 Dehydrogenation
Authors:
Chaoqun Li,
Weijie Yang,
Hao Liu,
Xinyuan Liu,
Xiujing Xing,
Zhengyang Gao,
Shuai Dong,
Hao Li
Abstract:
Developing solid-state hydrogen storage materials is as pressing as ever, which requires a comprehensive understanding of the dehydrogenation chemistry of a solid-state hydride. Transition state search and kinetics calculations are essential to understanding and designing high-performance solid-state hydrogen storage materials by filling in the knowledge gap that current experimental techniques ca…
▽ More
Developing solid-state hydrogen storage materials is as pressing as ever, which requires a comprehensive understanding of the dehydrogenation chemistry of a solid-state hydride. Transition state search and kinetics calculations are essential to understanding and designing high-performance solid-state hydrogen storage materials by filling in the knowledge gap that current experimental techniques cannot measure. However, the ab initio analysis of these processes is computationally expensive and time-consuming. Searching for descriptors to accurately predict the energy barrier is urgently needed, to accelerate the prediction of hydrogen storage material properties and identify the opportunities and challenges in this field. Herein, we develop a data-driven model to describe and predict the dehydrogenation barriers of a typical solid-state hydrogen storage material, magnesium hydride (MgH2), based on the combination of the crystal Hamilton population orbital of Mg-H bond and the distance between atomic hydrogen. By deriving the distance energy ratio, this model elucidates the key chemistry of the reaction kinetics. All the parameters in this model can be directly calculated with significantly less computational cost than conventional transition state search, so that the dehydrogenation performance of hydrogen storage materials can be predicted efficiently. Finally, we found that this model leads to excellent agreement with typical experimental measurements reported to date and provides clear design guidelines on how to propel the performance of MgH2 closer to the target set by the United States Department of Energy (US-DOE).
△ Less
Submitted 29 April, 2024; v1 submitted 15 April, 2024;
originally announced April 2024.
-
Ultralarge polarization in ferroelectric hafnia-based thin films
Authors:
Han Wu,
Kun Lin,
Qinghua Zhang,
Qian Yu,
Xiaoqian Fu,
Qiang Li,
Meera Cheviri,
Oswaldo Dieguez,
Shuai Xu,
Lin Gu,
Yili Cao,
Jiaou Wang,
Zhen Wang,
Yu Chen,
Huanhua Wang,
Jinxia Deng,
Jun Miao,
Xianran Xing
Abstract:
Hafnia-based ferroelectrics have become a valuable class of electronic functional materials at the nanoscale, showing great potential for next-generation memory and logic devices. However, more robust ferroelectric properties and better understanding of the polarization mechanisms are currently needed both in technology and science. Herein, we report the properties of oxygen-deficient Hf0.5Zr0.5O2…
▽ More
Hafnia-based ferroelectrics have become a valuable class of electronic functional materials at the nanoscale, showing great potential for next-generation memory and logic devices. However, more robust ferroelectric properties and better understanding of the polarization mechanisms are currently needed both in technology and science. Herein, we report the properties of oxygen-deficient Hf0.5Zr0.5O2 films with ultralarge remanent polarization (Pr) of 387 uC cm-2 at room temperature (1 kHz). Structure characterizations identify a new ferroelectric monoclinic Pc phase in these Hf0.5Zr0.5O2 films. The in-situ STEM measurements evidence polar displacements of the oxygen atoms, which move up and down in the Pc structure under applied DC bias fields, showing a huge displacement (1.6 A). DFT calculations optimized the Pc structure and also predicted a large polarization. The coexistence of the ferroelectric monoclinic (Pc) phases and orthorhombic (Pca21) is responsible for this superior ferroelectric properties. These findings are promising for hafnia-based ferroelectric applications in integrated ferroelectric devices, energy harvesting and actuators, etc.
△ Less
Submitted 18 March, 2024;
originally announced March 2024.
-
Towards Zero-shot Human-Object Interaction Detection via Vision-Language Integration
Authors:
Weiying Xue,
Qi Liu,
Qiwei Xiong,
Yuxiao Wang,
Zhenao Wei,
Xiaofen Xing,
Xiangmin Xu
Abstract:
Human-object interaction (HOI) detection aims to locate human-object pairs and identify their interaction categories in images. Most existing methods primarily focus on supervised learning, which relies on extensive manual HOI annotations. In this paper, we propose a novel framework, termed Knowledge Integration to HOI (KI2HOI), that effectively integrates the knowledge of visual-language model to…
▽ More
Human-object interaction (HOI) detection aims to locate human-object pairs and identify their interaction categories in images. Most existing methods primarily focus on supervised learning, which relies on extensive manual HOI annotations. In this paper, we propose a novel framework, termed Knowledge Integration to HOI (KI2HOI), that effectively integrates the knowledge of visual-language model to improve zero-shot HOI detection. Specifically, the verb feature learning module is designed based on visual semantics, by employing the verb extraction decoder to convert corresponding verb queries into interaction-specific category representations. We develop an effective additive self-attention mechanism to generate more comprehensive visual representations. Moreover, the innovative interaction representation decoder effectively extracts informative regions by integrating spatial and visual feature information through a cross-attention mechanism. To deal with zero-shot learning in low-data, we leverage a priori knowledge from the CLIP text encoder to initialize the linear classifier for enhanced interaction understanding. Extensive experiments conducted on the mainstream HICO-DET and V-COCO datasets demonstrate that our model outperforms the previous methods in various zero-shot and full-supervised settings.
△ Less
Submitted 11 March, 2024;
originally announced March 2024.
-
FinReport: Explainable Stock Earnings Forecasting via News Factor Analyzing Model
Authors:
Xiangyu Li,
Xinjie Shen,
Yawen Zeng,
Xiaofen Xing,
Jin Xu
Abstract:
The task of stock earnings forecasting has received considerable attention due to the demand investors in real-world scenarios. However, compared with financial institutions, it is not easy for ordinary investors to mine factors and analyze news. On the other hand, although large language models in the financial field can serve users in the form of dialogue robots, it still requires users to have…
▽ More
The task of stock earnings forecasting has received considerable attention due to the demand investors in real-world scenarios. However, compared with financial institutions, it is not easy for ordinary investors to mine factors and analyze news. On the other hand, although large language models in the financial field can serve users in the form of dialogue robots, it still requires users to have financial knowledge to ask reasonable questions. To serve the user experience, we aim to build an automatic system, FinReport, for ordinary investors to collect information, analyze it, and generate reports after summarizing.
Specifically, our FinReport is based on financial news announcements and a multi-factor model to ensure the professionalism of the report. The FinReport consists of three modules: news factorization module, return forecasting module, risk assessment module. The news factorization module involves understanding news information and combining it with stock factors, the return forecasting module aim to analysis the impact of news on market sentiment, and the risk assessment module is adopted to control investment risk. Extensive experiments on real-world datasets have well verified the effectiveness and explainability of our proposed FinReport. Our codes and datasets are available at https://github.com/frinkleko/FinReport.
△ Less
Submitted 4 March, 2024;
originally announced March 2024.
-
PointCore: Efficient Unsupervised Point Cloud Anomaly Detector Using Local-Global Features
Authors:
Baozhu Zhao,
Qiwei Xiong,
Xiaohan Zhang,
Jingfeng Guo,
Qi Liu,
Xiaofen Xing,
Xiangmin Xu
Abstract:
Three-dimensional point cloud anomaly detection that aims to detect anomaly data points from a training set serves as the foundation for a variety of applications, including industrial inspection and autonomous driving. However, existing point cloud anomaly detection methods often incorporate multiple feature memory banks to fully preserve local and global representations, which comes at the high…
▽ More
Three-dimensional point cloud anomaly detection that aims to detect anomaly data points from a training set serves as the foundation for a variety of applications, including industrial inspection and autonomous driving. However, existing point cloud anomaly detection methods often incorporate multiple feature memory banks to fully preserve local and global representations, which comes at the high cost of computational complexity and mismatches between features. To address that, we propose an unsupervised point cloud anomaly detection framework based on joint local-global features, termed PointCore. To be specific, PointCore only requires a single memory bank to store local (coordinate) and global (PointMAE) representations and different priorities are assigned to these local-global features, thereby reducing the computational cost and mismatching disturbance in inference. Furthermore, to robust against the outliers, a normalization ranking method is introduced to not only adjust values of different scales to a notionally common scale, but also transform densely-distributed data into a uniform distribution. Extensive experiments on Real3D-AD dataset demonstrate that PointCore achieves competitive inference time and the best performance in both detection and localization as compared to the state-of-the-art Reg3D-AD approach and several competitors.
△ Less
Submitted 4 March, 2024;
originally announced March 2024.
-
The Staggered Mesh Method: Accurate Exact Exchange towards the Thermodynamic Limit for Solids
Authors:
Stephen Jon Quiton,
Hamlin Wu,
Xin Xing,
Lin Lin,
Martin Head-Gordon
Abstract:
In periodic systems, the Hartree-Fock (HF) exchange energy exhibits the slowest convergence of all HF energy components as the system size approaches the thermodynamic limit. We demonstrate that the recently proposed staggered mesh method for Fock exchange energy [Xing, Li, and Lin, Math. Comp., 2024], which is specifically designed to sidestep certain singularities in exchange energy evaluation,…
▽ More
In periodic systems, the Hartree-Fock (HF) exchange energy exhibits the slowest convergence of all HF energy components as the system size approaches the thermodynamic limit. We demonstrate that the recently proposed staggered mesh method for Fock exchange energy [Xing, Li, and Lin, Math. Comp., 2024], which is specifically designed to sidestep certain singularities in exchange energy evaluation, can expedite the finite-size convergence rate for the exact exchange energy across a range of insulators and semiconductors when compared to the regular and truncated Coulomb methods. This remains true even for two computationally cheaper versions of this new method, which we call Non-SCF and Split-SCF staggered mesh. Additionally, a sequence of numerical tests on simple solids showcases the staggered mesh method's ability to improve convergence towards the thermodynamic limit for band gaps, bulk moduli, equilibrium lattice dimensions, energies, and phonon force constants.
△ Less
Submitted 21 February, 2024;
originally announced February 2024.
-
A spin-torque nano-oscillator based on interlayer-coupled meron-skyrmion pairs with a fixed orbit
Authors:
Qiyun Yi,
Ting Han,
Jinyi Jiang,
Xiangjun Xing
Abstract:
In recent years, magnetic skyrmion-based spin-torque nano-oscillators (STNOs) attract considerable interest for their prospect in future-generation communication and spintronic technologies. However, some critical issues, which hamper their practical applications, e.g., the long start-up time and variable skyrmion gyration orbit, remain to be resolved. Here, we numerically demonstrate a realizatio…
▽ More
In recent years, magnetic skyrmion-based spin-torque nano-oscillators (STNOs) attract considerable interest for their prospect in future-generation communication and spintronic technologies. However, some critical issues, which hamper their practical applications, e.g., the long start-up time and variable skyrmion gyration orbit, remain to be resolved. Here, we numerically demonstrate a realization of a fixed-orbit STNO, which is based on an interlayer-coupled meron-skyrmion (MS) pair other than a magnetic skyrmion. In this STNO, the MS pair possesses a structurally defined, fixed orbit within a broad range of driving current, even in the presence of random defects. The output frequency range of the STNO based on an MS pair far exceeds that of the STNO typically based on a single skyrmion. Moreover, the output frequency of this STNO can be further elevated if more MS pairs are incorporated. Our results reveal the nontrivial dynamics of the interlayer-coupled MS pair, opening perspectives for the design and optimization of fundamental spintronic devices.
△ Less
Submitted 9 February, 2024;
originally announced February 2024.
-
Assessing the Efficacy of Invisible Watermarks in AI-Generated Medical Images
Authors:
Xiaodan Xing,
Huiyu Zhou,
Yingying Fang,
Guang Yang
Abstract:
AI-generated medical images are gaining growing popularity due to their potential to address the data scarcity challenge in the real world. However, the issue of accurate identification of these synthetic images, particularly when they exhibit remarkable realism with their real copies, remains a concern. To mitigate this challenge, image generators such as DALLE and Imagen, have integrated digital…
▽ More
AI-generated medical images are gaining growing popularity due to their potential to address the data scarcity challenge in the real world. However, the issue of accurate identification of these synthetic images, particularly when they exhibit remarkable realism with their real copies, remains a concern. To mitigate this challenge, image generators such as DALLE and Imagen, have integrated digital watermarks aimed at facilitating the discernment of synthetic images' authenticity. These watermarks are embedded within the image pixels and are invisible to the human eye while remains their detectability. Nevertheless, a comprehensive investigation into the potential impact of these invisible watermarks on the utility of synthetic medical images has been lacking. In this study, we propose the incorporation of invisible watermarks into synthetic medical images and seek to evaluate their efficacy in the context of downstream classification tasks. Our goal is to pave the way for discussions on the viability of such watermarks in boosting the detectability of synthetic medical images, fortifying ethical standards, and safeguarding against data pollution and potential scams.
△ Less
Submitted 21 May, 2024; v1 submitted 5 February, 2024;
originally announced February 2024.
-
Measurements of Normalized Differential Cross Sections of Inclusive $η$ Production in $e^{+}e^{-}$ Annihilation at Energy from 2.0000 to 3.6710 GeV
Authors:
BESIII Collaboration,
M. Ablikim,
M. N. Achasov,
P. Adlarson,
O. Afedulidis,
X. C. Ai,
R. Aliberti,
A. Amoroso,
Q. An,
D. Anderle,
Y. Bai,
O. Bakina,
I. Balossino,
Y. Ban,
H. -R. Bao,
V. Batozskaya,
K. Begzsuren,
N. Berger,
M. Berlowski,
M. Bertani,
D. Bettoni,
F. Bianchi,
E. Bianco,
A. Bortone,
I. Boyko
, et al. (641 additional authors not shown)
Abstract:
Using data samples collected with the BESIII detector operating at the BEPCII storage ring, the cross section of the inclusive process $e^{+}e^{-} \to η+ X$, normalized by the total cross section of $e^{+}e^{-} \to \text{hadrons}$, is measured at eight center-of-mass energy points from 2.0000 GeV to 3.6710 GeV. These are the first measurements with momentum dependence in this energy region. Our me…
▽ More
Using data samples collected with the BESIII detector operating at the BEPCII storage ring, the cross section of the inclusive process $e^{+}e^{-} \to η+ X$, normalized by the total cross section of $e^{+}e^{-} \to \text{hadrons}$, is measured at eight center-of-mass energy points from 2.0000 GeV to 3.6710 GeV. These are the first measurements with momentum dependence in this energy region. Our measurement shows a significant discrepancy from calculations with the existing fragmentation functions. To address this discrepancy, a new QCD analysis is performed at the next-to-next-to-leading order with hadron mass corrections and higher twist effects, which can explain both the established high-energy data and our measurements reasonably well.
△ Less
Submitted 15 July, 2024; v1 submitted 31 January, 2024;
originally announced January 2024.
-
Unusually weak irradiation effects in anisotropic iron-based superconductor RbCa2Fe4As4F2
Authors:
Daniele Torsello,
Erik Piatti,
Michela Fracasso,
Roberto Gerbaldo,
Laura Gozzelino,
Xiaolei Yi,
Xiangzhuo Xing,
Zhixiang Shi,
Dario Daghero,
Gianluca Ghigo
Abstract:
We report on the effects of 3.5 MeV proton irradiation in RbCa$_2$Fe$_4$As$_4$F$_2$, an iron-based superconductor with unusual properties in between those of the pnictides and of the cuprate high-temperature superconductors. We studied how structural disorder introduced by ion bombardment affects the critical temperature, superfluid density and gap values by combining a coplanar waveguide resonato…
▽ More
We report on the effects of 3.5 MeV proton irradiation in RbCa$_2$Fe$_4$As$_4$F$_2$, an iron-based superconductor with unusual properties in between those of the pnictides and of the cuprate high-temperature superconductors. We studied how structural disorder introduced by ion bombardment affects the critical temperature, superfluid density and gap values by combining a coplanar waveguide resonator technique, electric transport measurements and point-contact Andreev-reflection spectroscopy. We find an unusually weak dependence of the superconducting properties on the amount of disorder in this material when compared to other iron-based superconductors under comparable irradiation conditions. The nodal multigap state exhibited by pristine RbCa$_2$Fe$_4$As$_4$F$_2$ is also robust against proton irradiation, with a two-band $d-d$ model being the one that best fits the experimental data.
△ Less
Submitted 16 January, 2024;
originally announced January 2024.
-
A clean-label graph backdoor attack method in node classification task
Authors:
Xiaogang Xing,
Ming Xu,
Yujing Bai,
Dongdong Yang
Abstract:
Backdoor attacks in the traditional graph neural networks (GNNs) field are easily detectable due to the dilemma of confusing labels. To explore the backdoor vulnerability of GNNs and create a more stealthy backdoor attack method, a clean-label graph backdoor attack method(CGBA) in the node classification task is proposed in this paper. Differently from existing backdoor attack methods, CGBA requir…
▽ More
Backdoor attacks in the traditional graph neural networks (GNNs) field are easily detectable due to the dilemma of confusing labels. To explore the backdoor vulnerability of GNNs and create a more stealthy backdoor attack method, a clean-label graph backdoor attack method(CGBA) in the node classification task is proposed in this paper. Differently from existing backdoor attack methods, CGBA requires neither modification of node labels nor graph structure. Specifically, to solve the problem of inconsistency between the contents and labels of the samples, CGBA selects poisoning samples in a specific target class and uses the label of sample as the target label (i.e., clean-label) after injecting triggers into the target samples. To guarantee the similarity of neighboring nodes, the raw features of the nodes are elaborately picked as triggers to further improve the concealment of the triggers. Extensive experiments results show the effectiveness of our method. When the poisoning rate is 0.04, CGBA can achieve an average attack success rate of 87.8%, 98.9%, 89.1%, and 98.5%, respectively.
△ Less
Submitted 30 December, 2023;
originally announced January 2024.
-
SVGDreamer: Text Guided SVG Generation with Diffusion Model
Authors:
Ximing Xing,
Haitao Zhou,
Chuang Wang,
Jing Zhang,
Dong Xu,
Qian Yu
Abstract:
Recently, text-guided scalable vector graphics (SVGs) synthesis has shown promise in domains such as iconography and sketch. However, existing text-to-SVG generation methods lack editability and struggle with visual quality and result diversity. To address these limitations, we propose a novel text-guided vector graphics synthesis method called SVGDreamer. SVGDreamer incorporates a semantic-driven…
▽ More
Recently, text-guided scalable vector graphics (SVGs) synthesis has shown promise in domains such as iconography and sketch. However, existing text-to-SVG generation methods lack editability and struggle with visual quality and result diversity. To address these limitations, we propose a novel text-guided vector graphics synthesis method called SVGDreamer. SVGDreamer incorporates a semantic-driven image vectorization (SIVE) process that enables the decomposition of synthesis into foreground objects and background, thereby enhancing editability. Specifically, the SIVE process introduces attention-based primitive control and an attention-mask loss function for effective control and manipulation of individual elements. Additionally, we propose a Vectorized Particle-based Score Distillation (VPSD) approach to address issues of shape over-smoothing, color over-saturation, limited diversity, and slow convergence of the existing text-to-SVG generation methods by modeling SVGs as distributions of control points and colors. Furthermore, VPSD leverages a reward model to re-weight vector particles, which improves aesthetic appeal and accelerates convergence. Extensive experiments are conducted to validate the effectiveness of SVGDreamer, demonstrating its superiority over baseline methods in terms of editability, visual quality, and diversity. Project page: \href{https://ximinng.github.io/SVGDreamer-project/}{https://ximinng.github.io/SVGDreamer-project/}
△ Less
Submitted 2 April, 2024; v1 submitted 27 December, 2023;
originally announced December 2023.
-
Hunting imaging biomarkers in pulmonary fibrosis: Benchmarks of the AIIB23 challenge
Authors:
Yang Nan,
Xiaodan Xing,
Shiyi Wang,
Zeyu Tang,
Federico N Felder,
Sheng Zhang,
Roberta Eufrasia Ledda,
Xiaoliu Ding,
Ruiqi Yu,
Weiping Liu,
Feng Shi,
Tianyang Sun,
Zehong Cao,
Minghui Zhang,
Yun Gu,
Hanxiao Zhang,
Jian Gao,
Pingyu Wang,
Wen Tang,
Pengxin Yu,
Han Kang,
Junqiang Chen,
Xing Lu,
Boyu Zhang,
Michail Mamalakis
, et al. (16 additional authors not shown)
Abstract:
Airway-related quantitative imaging biomarkers are crucial for examination, diagnosis, and prognosis in pulmonary diseases. However, the manual delineation of airway trees remains prohibitively time-consuming. While significant efforts have been made towards enhancing airway modelling, current public-available datasets concentrate on lung diseases with moderate morphological variations. The intric…
▽ More
Airway-related quantitative imaging biomarkers are crucial for examination, diagnosis, and prognosis in pulmonary diseases. However, the manual delineation of airway trees remains prohibitively time-consuming. While significant efforts have been made towards enhancing airway modelling, current public-available datasets concentrate on lung diseases with moderate morphological variations. The intricate honeycombing patterns present in the lung tissues of fibrotic lung disease patients exacerbate the challenges, often leading to various prediction errors. To address this issue, the 'Airway-Informed Quantitative CT Imaging Biomarker for Fibrotic Lung Disease 2023' (AIIB23) competition was organized in conjunction with the official 2023 International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI). The airway structures were meticulously annotated by three experienced radiologists. Competitors were encouraged to develop automatic airway segmentation models with high robustness and generalization abilities, followed by exploring the most correlated QIB of mortality prediction. A training set of 120 high-resolution computerised tomography (HRCT) scans were publicly released with expert annotations and mortality status. The online validation set incorporated 52 HRCT scans from patients with fibrotic lung disease and the offline test set included 140 cases from fibrosis and COVID-19 patients. The results have shown that the capacity of extracting airway trees from patients with fibrotic lung disease could be enhanced by introducing voxel-wise weighted general union loss and continuity loss. In addition to the competitive image biomarkers for prognosis, a strong airway-derived biomarker (Hazard ratio>1.5, p<0.0001) was revealed for survival prognostication compared with existing clinical measurements, clinician assessment and AI-based biomarkers.
△ Less
Submitted 16 April, 2024; v1 submitted 21 December, 2023;
originally announced December 2023.
-
A novel diffusion recommendation algorithm based on multi-scale cnn and residual lstm
Authors:
Yong Niu,
Xing Xing,
Zhichun Jia,
Ruidi Liu,
Mindong Xin
Abstract:
Sequential recommendation aims to infer user preferences from historical interaction sequences and predict the next item that users may be interested in the future. The current mainstream design approach is to represent items as fixed vectors, capturing the underlying relationships between items and user preferences based on the order of interactions. However, relying on a single fixed-item embedd…
▽ More
Sequential recommendation aims to infer user preferences from historical interaction sequences and predict the next item that users may be interested in the future. The current mainstream design approach is to represent items as fixed vectors, capturing the underlying relationships between items and user preferences based on the order of interactions. However, relying on a single fixed-item embedding may weaken the modeling capability of the system, and the global dynamics and local saliency exhibited by user preferences need to be distinguished. To address these issues, this paper proposes a novel diffusion recommendation algorithm based on multi-scale cnn and residual lstm (AREAL). We introduce diffusion models into the recommend system, representing items as probability distributions instead of fixed vectors. This approach enables adaptive reflection of multiple aspects of the items and generates item distributions in a denoising manner. We use multi-scale cnn and residual lstm methods to extract the local and global dependency features of user history interactions, and use attention mechanism to distinguish weights as the guide features of reverse diffusion recovery. The effectiveness of the proposed method is validated through experiments conducted on two real-world datasets. Specifically, AREAL obtains improvements over the best baselines by 2.63% and 4.25% in terms of HR@20 and 5.05% and 3.94% in terms of NDCG@20 on all datasets.
△ Less
Submitted 20 December, 2023; v1 submitted 17 December, 2023;
originally announced December 2023.
-
BiPFT: Binary Pre-trained Foundation Transformer with Low-rank Estimation of Binarization Residual Polynomials
Authors:
Xingrun Xing,
Li Du,
Xinyuan Wang,
Xianlin Zeng,
Yequan Wang,
Zheng Zhang,
Jiajun Zhang
Abstract:
Pretrained foundation models offer substantial benefits for a wide range of downstream tasks, which can be one of the most potential techniques to access artificial general intelligence. However, scaling up foundation transformers for maximal task-agnostic knowledge has brought about computational challenges, especially on resource-limited devices such as mobiles. This work proposes the first Bina…
▽ More
Pretrained foundation models offer substantial benefits for a wide range of downstream tasks, which can be one of the most potential techniques to access artificial general intelligence. However, scaling up foundation transformers for maximal task-agnostic knowledge has brought about computational challenges, especially on resource-limited devices such as mobiles. This work proposes the first Binary Pretrained Foundation Transformer (BiPFT) for natural language understanding (NLU) tasks, which remarkably saves 56 times operations and 28 times memory. In contrast to previous task-specific binary transformers, BiPFT exhibits a substantial enhancement in the learning capabilities of binary neural networks (BNNs), promoting BNNs into the era of pre-training. Benefiting from extensive pretraining data, we further propose a data-driven binarization method. Specifically, we first analyze the binarization error in self-attention operations and derive the polynomials of binarization error. To simulate full-precision self-attention, we define binarization error as binarization residual polynomials, and then introduce low-rank estimators to model these polynomials. Extensive experiments validate the effectiveness of BiPFTs, surpassing task-specific baseline by 15.4% average performance on the GLUE benchmark. BiPFT also demonstrates improved robustness to hyperparameter changes, improved optimization efficiency, and reduced reliance on downstream distillation, which consequently generalize on various NLU tasks and simplify the downstream pipeline of BNNs. Our code and pretrained models are publicly available at https://github.com/Xingrun-Xing/BiPFT.
△ Less
Submitted 20 June, 2024; v1 submitted 14 December, 2023;
originally announced December 2023.
-
LD-SDM: Language-Driven Hierarchical Species Distribution Modeling
Authors:
Srikumar Sastry,
Xin Xing,
Aayush Dhakal,
Subash Khanal,
Adeel Ahmad,
Nathan Jacobs
Abstract:
We focus on the problem of species distribution modeling using global-scale presence-only data. Most previous studies have mapped the range of a given species using geographical and environmental features alone. To capture a stronger implicit relationship between species, we encode the taxonomic hierarchy of species using a large language model. This enables range mapping for any taxonomic rank an…
▽ More
We focus on the problem of species distribution modeling using global-scale presence-only data. Most previous studies have mapped the range of a given species using geographical and environmental features alone. To capture a stronger implicit relationship between species, we encode the taxonomic hierarchy of species using a large language model. This enables range mapping for any taxonomic rank and unseen species without additional supervision. Further, we propose a novel proximity-aware evaluation metric that enables evaluating species distribution models using any pixel-level representation of ground-truth species range map. The proposed metric penalizes the predictions of a model based on its proximity to the ground truth. We describe the effectiveness of our model by systematically evaluating on the task of species range prediction, zero-shot prediction and geo-feature regression against the state-of-the-art. Results show our model outperforms the strong baselines when trained with a variety of multi-label learning losses.
△ Less
Submitted 13 December, 2023;
originally announced December 2023.