-
Make a Strong Teacher with Label Assistance: A Novel Knowledge Distillation Approach for Semantic Segmentation
Authors:
Shoumeng Qiu,
Jie Chen,
Xinrun Li,
Ru Wan,
Xiangyang Xue,
Jian Pu
Abstract:
In this paper, we introduce a novel knowledge distillation approach for the semantic segmentation task. Unlike previous methods that rely on power-trained teachers or other modalities to provide additional knowledge, our approach does not require complex teacher models or information from extra sensors. Specifically, for the teacher model training, we propose to noise the label and then incorporat…
▽ More
In this paper, we introduce a novel knowledge distillation approach for the semantic segmentation task. Unlike previous methods that rely on power-trained teachers or other modalities to provide additional knowledge, our approach does not require complex teacher models or information from extra sensors. Specifically, for the teacher model training, we propose to noise the label and then incorporate it into input to effectively boost the lightweight teacher performance. To ensure the robustness of the teacher model against the introduced noise, we propose a dual-path consistency training strategy featuring a distance loss between the outputs of two paths. For the student model training, we keep it consistent with the standard distillation for simplicity. Our approach not only boosts the efficacy of knowledge distillation but also increases the flexibility in selecting teacher and student models. To demonstrate the advantages of our Label Assisted Distillation (LAD) method, we conduct extensive experiments on five challenging datasets including Cityscapes, ADE20K, PASCAL-VOC, COCO-Stuff 10K, and COCO-Stuff 164K, five popular models: FCN, PSPNet, DeepLabV3, STDC, and OCRNet, and results show the effectiveness and generalization of our approach. We posit that incorporating labels into the input, as demonstrated in our work, will provide valuable insights into related fields. Code is available at https://github.com/skyshoumeng/Label_Assisted_Distillation.
△ Less
Submitted 18 July, 2024;
originally announced July 2024.
-
Automated Label Unification for Multi-Dataset Semantic Segmentation with GNNs
Authors:
Rong Ma,
Jie Chen,
Xiangyang Xue,
Jian Pu
Abstract:
Deep supervised models possess significant capability to assimilate extensive training data, thereby presenting an opportunity to enhance model performance through training on multiple datasets. However, conflicts arising from different label spaces among datasets may adversely affect model performance. In this paper, we propose a novel approach to automatically construct a unified label space acr…
▽ More
Deep supervised models possess significant capability to assimilate extensive training data, thereby presenting an opportunity to enhance model performance through training on multiple datasets. However, conflicts arising from different label spaces among datasets may adversely affect model performance. In this paper, we propose a novel approach to automatically construct a unified label space across multiple datasets using graph neural networks. This enables semantic segmentation models to be trained simultaneously on multiple datasets, resulting in performance improvements. Unlike existing methods, our approach facilitates seamless training without the need for additional manual reannotation or taxonomy reconciliation. This significantly enhances the efficiency and effectiveness of multi-dataset segmentation model training. The results demonstrate that our method significantly outperforms other multi-dataset training methods when trained on seven datasets simultaneously, and achieves state-of-the-art performance on the WildDash 2 benchmark.
△ Less
Submitted 15 July, 2024;
originally announced July 2024.
-
MambaForGCN: Enhancing Long-Range Dependency with State Space Model and Kolmogorov-Arnold Networks for Aspect-Based Sentiment Analysis
Authors:
Adamu Lawan,
Juhua Pu,
Haruna Yunusa,
Aliyu Umar,
Muhammad Lawan
Abstract:
Aspect-based sentiment Analysis (ABSA) identifies and evaluates sentiments toward specific aspects of entities within text, providing detailed insights beyond overall sentiment. However, Attention mechanisms and neural network models struggle with syntactic constraints, and the quadratic complexity of attention mechanisms hinders their adoption for capturing long-range dependencies between aspect…
▽ More
Aspect-based sentiment Analysis (ABSA) identifies and evaluates sentiments toward specific aspects of entities within text, providing detailed insights beyond overall sentiment. However, Attention mechanisms and neural network models struggle with syntactic constraints, and the quadratic complexity of attention mechanisms hinders their adoption for capturing long-range dependencies between aspect and opinion words in ABSA. This complexity can lead to the misinterpretation of irrelevant con-textual words, restricting their effectiveness to short-range dependencies. Some studies have investigated merging semantic and syntactic approaches but face challenges in effectively integrating these methods. To address the above problems, we present MambaForGCN, a novel approach to enhance short and long-range dependencies between aspect and opinion words in ABSA. This innovative approach incorporates syntax-based Graph Convolutional Network (SynGCN) and MambaFormer (Mamba-Transformer) modules to encode input with dependency relations and semantic information. The Multihead Attention (MHA) and Mamba blocks in the MambaFormer module serve as channels to enhance the model with short and long-range dependencies between aspect and opinion words. We also introduce the Kolmogorov-Arnold Networks (KANs) gated fusion, an adaptively integrated feature representation system combining SynGCN and MambaFormer representations. Experimental results on three benchmark datasets demonstrate MambaForGCN's effectiveness, outperforming state-of-the-art (SOTA) baseline models.
△ Less
Submitted 14 July, 2024;
originally announced July 2024.
-
How to Make Cross Encoder a Good Teacher for Efficient Image-Text Retrieval?
Authors:
Yuxin Chen,
Zongyang Ma,
Ziqi Zhang,
Zhongang Qi,
Chunfeng Yuan,
Bing Li,
Junfu Pu,
Ying Shan,
Xiaojuan Qi,
Weiming Hu
Abstract:
Dominant dual-encoder models enable efficient image-text retrieval but suffer from limited accuracy while the cross-encoder models offer higher accuracy at the expense of efficiency. Distilling cross-modality matching knowledge from cross-encoder to dual-encoder provides a natural approach to harness their strengths. Thus we investigate the following valuable question: how to make cross-encoder a…
▽ More
Dominant dual-encoder models enable efficient image-text retrieval but suffer from limited accuracy while the cross-encoder models offer higher accuracy at the expense of efficiency. Distilling cross-modality matching knowledge from cross-encoder to dual-encoder provides a natural approach to harness their strengths. Thus we investigate the following valuable question: how to make cross-encoder a good teacher for dual-encoder? Our findings are threefold:(1) Cross-modal similarity score distribution of cross-encoder is more concentrated while the result of dual-encoder is nearly normal making vanilla logit distillation less effective. However ranking distillation remains practical as it is not affected by the score distribution.(2) Only the relative order between hard negatives conveys valid knowledge while the order information between easy negatives has little significance.(3) Maintaining the coordination between distillation loss and dual-encoder training loss is beneficial for knowledge transfer. Based on these findings we propose a novel Contrastive Partial Ranking Distillation (CPRD) method which implements the objective of mimicking relative order between hard negative samples with contrastive learning. This approach coordinates with the training of the dual-encoder effectively transferring valid knowledge from the cross-encoder to the dual-encoder. Extensive experiments on image-text retrieval and ranking tasks show that our method surpasses other distillation methods and significantly improves the accuracy of dual-encoder.
△ Less
Submitted 10 July, 2024;
originally announced July 2024.
-
Rethinking Closed-loop Planning Framework for Imitation-based Model Integrating Prediction and Planning
Authors:
Jiayu Guo,
Mingyue Feng,
Pengfei Zhu,
Chengjun Li,
Jian Pu
Abstract:
In recent years, the integration of prediction and planning through neural networks has received substantial attention. Despite extensive studies on it, there is a noticeable gap in understanding the operation of such models within a closed-loop planning setting. To bridge this gap, we propose a novel closed-loop planning framework compatible with neural networks engaged in joint prediction and pl…
▽ More
In recent years, the integration of prediction and planning through neural networks has received substantial attention. Despite extensive studies on it, there is a noticeable gap in understanding the operation of such models within a closed-loop planning setting. To bridge this gap, we propose a novel closed-loop planning framework compatible with neural networks engaged in joint prediction and planning. The framework contains two running modes, namely planning and safety monitoring, wherein the neural network performs Motion Prediction and Planning (MPP) and Conditional Motion Prediction (CMP) correspondingly without altering architecture. We evaluate the efficacy of our framework using the nuPlan dataset and its simulator, conducting closed-loop experiments across diverse scenarios. The results demonstrate that the proposed framework ensures the feasibility and local stability of the planning process while maintaining safety with CMP safety monitoring. Compared to other learning-based methods, our approach achieves substantial improvement.
△ Less
Submitted 7 July, 2024;
originally announced July 2024.
-
Towards Open-set Camera 3D Object Detection
Authors:
Zhuolin He,
Xinrun Li,
Heng Gao,
Jiachen Tang,
Shoumeng Qiu,
Wenfu Wang,
Lvjian Lu,
Xuchong Qiu,
Xiangyang Xue,
Jian Pu
Abstract:
Traditional camera 3D object detectors are typically trained to recognize a predefined set of known object classes. In real-world scenarios, these detectors may encounter unknown objects outside the training categories and fail to identify them correctly. To address this gap, we present OS-Det3D (Open-set Camera 3D Object Detection), a two-stage training framework enhancing the ability of camera 3…
▽ More
Traditional camera 3D object detectors are typically trained to recognize a predefined set of known object classes. In real-world scenarios, these detectors may encounter unknown objects outside the training categories and fail to identify them correctly. To address this gap, we present OS-Det3D (Open-set Camera 3D Object Detection), a two-stage training framework enhancing the ability of camera 3D detectors to identify both known and unknown objects. The framework involves our proposed 3D Object Discovery Network (ODN3D), which is specifically trained using geometric cues such as the location and scale of 3D boxes to discover general 3D objects. ODN3D is trained in a class-agnostic manner, and the provided 3D object region proposals inherently come with data noise. To boost accuracy in identifying unknown objects, we introduce a Joint Objectness Selection (JOS) module. JOS selects the pseudo ground truth for unknown objects from the 3D object region proposals of ODN3D by combining the ODN3D objectness and camera feature attention objectness. Experiments on the nuScenes and KITTI datasets demonstrate the effectiveness of our framework in enabling camera 3D detectors to successfully identify unknown objects while also improving their performance on known objects.
△ Less
Submitted 26 June, 2024; v1 submitted 25 June, 2024;
originally announced June 2024.
-
OAML: Outlier Aware Metric Learning for OOD Detection Enhancement
Authors:
Heng Gao,
Zhuolin He,
Shoumeng Qiu,
Jian Pu
Abstract:
Out-of-distribution (OOD) detection methods have been developed to identify objects that a model has not seen during training. The Outlier Exposure (OE) methods use auxiliary datasets to train OOD detectors directly. However, the collection and learning of representative OOD samples may pose challenges. To tackle these issues, we propose the Outlier Aware Metric Learning (OAML) framework. The main…
▽ More
Out-of-distribution (OOD) detection methods have been developed to identify objects that a model has not seen during training. The Outlier Exposure (OE) methods use auxiliary datasets to train OOD detectors directly. However, the collection and learning of representative OOD samples may pose challenges. To tackle these issues, we propose the Outlier Aware Metric Learning (OAML) framework. The main idea of our method is to use the k-NN algorithm and Stable Diffusion model to generate outliers for training at the feature level without making any distributional assumptions. To increase feature discrepancies in the semantic space, we develop a mutual information-based contrastive learning approach for learning from OOD data effectively. Both theoretical and empirical results confirm the effectiveness of this contrastive learning technique. Furthermore, we incorporate knowledge distillation into our learning framework to prevent degradation of in-distribution classification accuracy. The combination of contrastive learning and knowledge distillation algorithms significantly enhances the performance of OOD detection. Experimental results across various datasets show that our method significantly outperforms previous OE methods.
△ Less
Submitted 24 June, 2024;
originally announced June 2024.
-
HoLLMwood: Unleashing the Creativity of Large Language Models in Screenwriting via Role Playing
Authors:
Jing Chen,
Xinyu Zhu,
Cheng Yang,
Chufan Shi,
Yadong Xi,
Yuxiang Zhang,
Junjie Wang,
Jiashu Pu,
Rongsheng Zhang,
Yujiu Yang,
Tian Feng
Abstract:
Generative AI has demonstrated unprecedented creativity in the field of computer vision, yet such phenomena have not been observed in natural language processing. In particular, large language models (LLMs) can hardly produce written works at the level of human experts due to the extremely high complexity of literature writing. In this paper, we present HoLLMwood, an automated framework for unleas…
▽ More
Generative AI has demonstrated unprecedented creativity in the field of computer vision, yet such phenomena have not been observed in natural language processing. In particular, large language models (LLMs) can hardly produce written works at the level of human experts due to the extremely high complexity of literature writing. In this paper, we present HoLLMwood, an automated framework for unleashing the creativity of LLMs and exploring their potential in screenwriting, which is a highly demanding task. Mimicking the human creative process, we assign LLMs to different roles involved in the real-world scenario. In addition to the common practice of treating LLMs as ${Writer}$, we also apply LLMs as ${Editor}$, who is responsible for providing feedback and revision advice to ${Writer}$. Besides, to enrich the characters and deepen the plots, we introduce a role-playing mechanism and adopt LLMs as ${Actors}$ that can communicate and interact with each other. Evaluations on automatically generated screenplays show that HoLLMwood substantially outperforms strong baselines in terms of coherence, relevance, interestingness and overall quality.
△ Less
Submitted 17 June, 2024;
originally announced June 2024.
-
Amplifying Aspect-Sentence Awareness: A Novel Approach for Aspect-Based Sentiment Analysis
Authors:
Adamu Lawan,
Juhua Pu,
Haruna Yunusa,
Jawad Muhammad,
Aliyu Umar
Abstract:
Aspect-Based Sentiment Analysis (ABSA) is increasingly crucial in Natural Language Processing (NLP) for applications such as customer feedback analysis and product recommendation systems. ABSA goes beyond traditional sentiment analysis by extracting sentiments related to specific aspects mentioned in the text; existing attention-based models often need help to effectively connect aspects with cont…
▽ More
Aspect-Based Sentiment Analysis (ABSA) is increasingly crucial in Natural Language Processing (NLP) for applications such as customer feedback analysis and product recommendation systems. ABSA goes beyond traditional sentiment analysis by extracting sentiments related to specific aspects mentioned in the text; existing attention-based models often need help to effectively connect aspects with context due to language complexity and multiple sentiment polarities in a single sentence. Recent research underscores the value of integrating syntactic information, such as dependency trees, to understand long-range syntactic relationships better and link aspects with context. Despite these advantages, challenges persist, including sensitivity to parsing errors and increased computational complexity when combining syntactic and semantic information. To address these issues, we propose Amplifying Aspect-Sentence Awareness (A3SN), a novel technique designed to enhance ABSA through amplifying aspect-sentence awareness attention. Following the transformer's standard process, our innovative approach incorporates multi-head attention mechanisms to augment the model with sentence and aspect semantic information. We added another multi-head attention module: amplify aspect-sentence awareness attention. By doubling its focus between the sentence and aspect, we effectively highlighted aspect importance within the sentence context. This enables accurate capture of subtle relationships and dependencies. Additionally, gated fusion integrates feature representations from multi-head and amplified aspect-sentence awareness attention mechanisms, which is essential for ABSA. Experimental results across three benchmark datasets demonstrate A3SN's effectiveness and outperform state-of-the-art (SOTA) baseline models.
△ Less
Submitted 14 May, 2024;
originally announced May 2024.
-
EC-SLAM: Real-time Dense Neural RGB-D SLAM System with Effectively Constrained Global Bundle Adjustment
Authors:
Guanghao Li,
Qi Chen,
YuXiang Yan,
Jian Pu
Abstract:
We introduce EC-SLAM, a real-time dense RGB-D simultaneous localization and mapping (SLAM) system utilizing Neural Radiance Fields (NeRF). Although recent NeRF-based SLAM systems have demonstrated encouraging outcomes, they have yet to completely leverage NeRF's capability to constrain pose optimization. By employing an effectively constrained global bundle adjustment (BA) strategy, our system mak…
▽ More
We introduce EC-SLAM, a real-time dense RGB-D simultaneous localization and mapping (SLAM) system utilizing Neural Radiance Fields (NeRF). Although recent NeRF-based SLAM systems have demonstrated encouraging outcomes, they have yet to completely leverage NeRF's capability to constrain pose optimization. By employing an effectively constrained global bundle adjustment (BA) strategy, our system makes use of NeRF's implicit loop closure correction capability. This improves the tracking accuracy by reinforcing the constraints on the keyframes that are most pertinent to the optimized current frame. In addition, by implementing a feature-based and uniform sampling strategy that minimizes the number of ineffective constraint points for pose optimization, we mitigate the effects of random sampling in NeRF. EC-SLAM utilizes sparse parametric encodings and the truncated signed distance field (TSDF) to represent the map in order to facilitate efficient fusion, resulting in reduced model parameters and accelerated convergence velocity. A comprehensive evaluation conducted on the Replica, ScanNet, and TUM datasets showcases cutting-edge performance, including enhanced reconstruction accuracy resulting from precise pose estimation, 21 Hz run time, and tracking precision improvements of up to 50\%. The source code is available at https://github.com/Lightingooo/EC-SLAM.
△ Less
Submitted 20 April, 2024;
originally announced April 2024.
-
Accelerating Regular Path Queries over Graph Database with Processing-in-Memory
Authors:
Ruoyan Ma,
Shengan Zheng,
Guifeng Wang,
Jin Pu,
Yifan Hua,
Wentao Wang,
Linpeng Huang
Abstract:
Regular path queries (RPQs) in graph databases are bottlenecked by the memory wall. Emerging processing-in-memory (PIM) technologies offer a promising solution to dispatch and execute path matching tasks in parallel within PIM modules. We present Moctopus, a PIM-based data management system for graph databases that supports efficient batch RPQs and graph updates. Moctopus employs a PIM-friendly dy…
▽ More
Regular path queries (RPQs) in graph databases are bottlenecked by the memory wall. Emerging processing-in-memory (PIM) technologies offer a promising solution to dispatch and execute path matching tasks in parallel within PIM modules. We present Moctopus, a PIM-based data management system for graph databases that supports efficient batch RPQs and graph updates. Moctopus employs a PIM-friendly dynamic graph partitioning algorithm, which tackles graph skewness and preserves graph locality with low overhead for RPQ processing. Moctopus enables efficient graph update by amortizing the host CPU's update overhead to PIM modules. Evaluation of Moctopus demonstrates superiority over the state-of-the-art traditional graph database.
△ Less
Submitted 15 March, 2024;
originally announced March 2024.
-
Pressure-enhanced $f$-electron orbital weighting in UTe2 mapped by quantum interferometry
Authors:
T. I. Weinberger,
Z. Wu,
A. J. Hickey,
D. E. Graf,
G. Li,
P. Wang,
R. Zhou,
A. Cabala,
J. Pu,
V. Sechovsky,
M. Valiska,
G. G. Lonzarich,
F. M. Grosche,
A. G. Eaton
Abstract:
The phase landscape of UTe$_2$ features a remarkable diversity of superconducting phases under applied pressure and magnetic field. Recent quantum oscillation studies at ambient pressure have revealed the quasi-2D Fermi surface of this material. However, the pressure-dependence of the Fermi surface remains an open question. Here we track the evolution of the UTe$_2$ Fermi surface as a function of…
▽ More
The phase landscape of UTe$_2$ features a remarkable diversity of superconducting phases under applied pressure and magnetic field. Recent quantum oscillation studies at ambient pressure have revealed the quasi-2D Fermi surface of this material. However, the pressure-dependence of the Fermi surface remains an open question. Here we track the evolution of the UTe$_2$ Fermi surface as a function of pressure up to 19.5 kbar by measuring quantum interference oscillations. We find that in sufficient magnetic field to suppress both superconductivity at low pressures and incommensurate antiferromagnetism at higher pressures, the quasi-2D Fermi surface found at ambient pressure smoothly connects to that at 19.5 kbar, with no signs of a reconstruction over this pressure interval. The warping of the cylindrical Fermi sheets continuously increases with pressure, which is consistent with increased $f$-orbital contribution at the Fermi level, up to and beyond the critical pressure at which superconductivity is truncated. These findings highlight the value of high pressure quantum interference measurements as a new probe of the electronic structure in heavy fermion materials.
△ Less
Submitted 6 March, 2024;
originally announced March 2024.
-
FastOcc: Accelerating 3D Occupancy Prediction by Fusing the 2D Bird's-Eye View and Perspective View
Authors:
Jiawei Hou,
Xiaoyan Li,
Wenhao Guan,
Gang Zhang,
Di Feng,
Yuheng Du,
Xiangyang Xue,
Jian Pu
Abstract:
In autonomous driving, 3D occupancy prediction outputs voxel-wise status and semantic labels for more comprehensive understandings of 3D scenes compared with traditional perception tasks, such as 3D object detection and bird's-eye view (BEV) semantic segmentation. Recent researchers have extensively explored various aspects of this task, including view transformation techniques, ground-truth label…
▽ More
In autonomous driving, 3D occupancy prediction outputs voxel-wise status and semantic labels for more comprehensive understandings of 3D scenes compared with traditional perception tasks, such as 3D object detection and bird's-eye view (BEV) semantic segmentation. Recent researchers have extensively explored various aspects of this task, including view transformation techniques, ground-truth label generation, and elaborate network design, aiming to achieve superior performance. However, the inference speed, crucial for running on an autonomous vehicle, is neglected. To this end, a new method, dubbed FastOcc, is proposed. By carefully analyzing the network effect and latency from four parts, including the input image resolution, image backbone, view transformation, and occupancy prediction head, it is found that the occupancy prediction head holds considerable potential for accelerating the model while keeping its accuracy. Targeted at improving this component, the time-consuming 3D convolution network is replaced with a novel residual-like architecture, where features are mainly digested by a lightweight 2D BEV convolution network and compensated by integrating the 3D voxel features interpolated from the original image features. Experiments on the Occ3D-nuScenes benchmark demonstrate that our FastOcc achieves state-of-the-art results with a fast inference speed.
△ Less
Submitted 5 March, 2024;
originally announced March 2024.
-
Crafting a Good Prompt or Providing Exemplary Dialogues? A Study of In-Context Learning for Persona-based Dialogue Generation
Authors:
Jiashu Pu,
Yajing Wan,
Yuru Zhang,
Jing Chen,
Ling Cheng,
Qian Shao,
Yongzhu Chang,
Tangjie Lv,
Rongsheng Zhang
Abstract:
Previous in-context learning (ICL) research has focused on tasks such as classification, machine translation, text2table, etc., while studies on whether ICL can improve human-like dialogue generation are scarce. Our work fills this gap by systematically investigating the ICL capabilities of large language models (LLMs) in persona-based dialogue generation, conducting extensive experiments on high-…
▽ More
Previous in-context learning (ICL) research has focused on tasks such as classification, machine translation, text2table, etc., while studies on whether ICL can improve human-like dialogue generation are scarce. Our work fills this gap by systematically investigating the ICL capabilities of large language models (LLMs) in persona-based dialogue generation, conducting extensive experiments on high-quality real human Chinese dialogue datasets. From experimental results, we draw three conclusions: 1) adjusting prompt instructions is the most direct, effective, and economical way to improve generation quality; 2) randomly retrieving demonstrations (demos) achieves the best results, possibly due to the greater diversity and the amount of effective information; counter-intuitively, retrieving demos with a context identical to the query performs the worst; 3) even when we destroy the multi-turn associations and single-turn semantics in the demos, increasing the number of demos still improves dialogue performance, proving that LLMs can learn from corrupted dialogue demos. Previous explanations of the ICL mechanism, such as $n$-gram induction head, cannot fully account for this phenomenon.
△ Less
Submitted 17 February, 2024; v1 submitted 15 February, 2024;
originally announced February 2024.
-
Causal Relationship Network of Risk Factors Impacting Workday Loss in Underground Coal Mines
Authors:
Shangsi Ren,
Cameron A. Beeche,
Zhiyi Shi,
Maria Acevedo Garcia,
Katherine Zychowski,
Shuguang Leng,
Pedram Roghanchi,
Jiantao Pu
Abstract:
This study aims to establish the causal relationship network between various factors leading to workday loss in underground coal mines using a novel causal artificial intelligence (AI) method. The analysis utilizes data obtained from the National Institute for Occupational Safety and Health (NIOSH). A total of 101,010 injury records from 3,982 unique underground coal mines spanning the years from…
▽ More
This study aims to establish the causal relationship network between various factors leading to workday loss in underground coal mines using a novel causal artificial intelligence (AI) method. The analysis utilizes data obtained from the National Institute for Occupational Safety and Health (NIOSH). A total of 101,010 injury records from 3,982 unique underground coal mines spanning the years from 1990 to 2020 were extracted from the NIOSH database. Causal relationships were analyzed and visualized using a novel causal AI method called Grouped Greedy Equivalence Search (GGES). The impact of each variable on workday loss was assessed through intervention do-calculus adjustment (IDA) scores. Model training and validation were performed using the 10-fold cross-validation technique. Performance metrics, including adjacency precision (AP), adjacency recall (AR), arrowhead precision (AHP), and arrowhead recall (AHR), were utilized to evaluate the models. Findings revealed that after 2006, key direct causes of workday loss among mining employees included total mining experience, mean office employees, mean underground employees, county, and total mining experience (years). Total mining experience emerged as the most influential factor, whereas mean employees per mine exhibited the least influence. The analyses emphasized the significant role of total mining experience in determining workday loss. The models achieved optimal performance, with AP, AR, AHP, and AHR values measuring 0.694, 0.653, 0.386, and 0.345, respectively. This study demonstrates the feasibility of utilizing the new GGES method to clarify the causal factors behind the workday loss by analyzing employment demographics and injury records and establish their causal relationship network.
△ Less
Submitted 24 January, 2024;
originally announced February 2024.
-
Probing the internal structures of $pΩ$ and $ΩΩ$ with their production at the LHC
Authors:
Jie Pu,
Kai-Jia Sun,
Chun-Wang Ma,
Lie-Wen Chen
Abstract:
The strange dibaryons $pΩ$ ($^5\rm{S}_2$) and $ΩΩ$ ($^1\rm{S}_0$) are likely bound, existing either in molecular states like the deuteron or as more exotic compact six-quark states. Here, we investigate the production of these two dibaryons in Pb+Pb collisions at $\sqrt{s_{NN}}$=2.76 TeV at the CERN Large Hadron Collider (LHC) within a covariant coalescence model, which employs a blast-wave-like p…
▽ More
The strange dibaryons $pΩ$ ($^5\rm{S}_2$) and $ΩΩ$ ($^1\rm{S}_0$) are likely bound, existing either in molecular states like the deuteron or as more exotic compact six-quark states. Here, we investigate the production of these two dibaryons in Pb+Pb collisions at $\sqrt{s_{NN}}$=2.76 TeV at the CERN Large Hadron Collider (LHC) within a covariant coalescence model, which employs a blast-wave-like parametrization for the phase-space configurations of constituent particles at freeze-out. For the molecular states, the $pΩ$ and $ΩΩ$ are produced via $p$-$Ω$ and $Ω$-$Ω$ coalescence, respectively, while for the six-quark states, they are formed through $uudsss$ and $ssssss$ coalescence. We find that the yield ratio $N_{pΩ}/N_Ω$ and $N_{ΩΩ}/N_Ω$ have a distinct centrality dependence between the molecular and multi-quark states, thus offering a promising way for distinguishing the two states. Our results suggest that the measurements of $pΩ$ and $ΩΩ$ production in relativistic heavy-ion collisions can shed light on their internal structures.
△ Less
Submitted 6 February, 2024;
originally announced February 2024.
-
Embedding Attack Project (Work Report)
Authors:
Jiameng Pu,
Zafar Takhirov
Abstract:
This report summarizes all the MIA experiments (Membership Inference Attacks) of the Embedding Attack Project, including threat models, experimental setup, experimental results, findings and discussion. Current results cover the evaluation of two main MIA strategies (loss-based and embedding-based MIAs) on 6 AI models ranging from Computer Vision to Language Modelling. There are two ongoing experi…
▽ More
This report summarizes all the MIA experiments (Membership Inference Attacks) of the Embedding Attack Project, including threat models, experimental setup, experimental results, findings and discussion. Current results cover the evaluation of two main MIA strategies (loss-based and embedding-based MIAs) on 6 AI models ranging from Computer Vision to Language Modelling. There are two ongoing experiments on MIA defense and neighborhood-comparison embedding attacks. These are ongoing projects.
The current work on MIA and PIA can be summarized into six conclusions: (1) Amount of overfitting is directly proportional to model's vulnerability; (2) early embedding layers in the model are less susceptible to privacy leaks; (3) Deeper model layers contain more membership information; (4) Models are more vulnerable to MIA if both embeddings and corresponding training labels are compromised; (5) it is possible to use pseudo-labels to increase the MIA success; and (6) although MIA and PIA success rates are proportional, reducing the MIA does not necessarily reduce the PIA.
△ Less
Submitted 24 January, 2024;
originally announced January 2024.
-
Lax pairs informed neural networks solving integrable systems
Authors:
Juncai Pu,
Yong Chen
Abstract:
Lax pairs are one of the most important features of integrable system. In this work, we propose the Lax pairs informed neural networks (LPNNs) tailored for the integrable systems with Lax pairs by designing novel network architectures and loss functions, comprising LPNN-v1 and LPNN-v2. The most noteworthy advantage of LPNN-v1 is that it can transform the solving of nonlinear integrable systems int…
▽ More
Lax pairs are one of the most important features of integrable system. In this work, we propose the Lax pairs informed neural networks (LPNNs) tailored for the integrable systems with Lax pairs by designing novel network architectures and loss functions, comprising LPNN-v1 and LPNN-v2. The most noteworthy advantage of LPNN-v1 is that it can transform the solving of nonlinear integrable systems into the solving of a linear Lax pairs spectral problems, and it not only efficiently solves data-driven localized wave solutions, but also obtains spectral parameter and corresponding spectral function in Lax pairs spectral problems of the integrable systems. On the basis of LPNN-v1, we additionally incorporate the compatibility condition/zero curvature equation of Lax pairs in LPNN-v2, its major advantage is the ability to solve and explore high-accuracy data-driven localized wave solutions and associated spectral problems for integrable systems with Lax pairs. The numerical experiments focus on studying abundant localized wave solutions for very important and representative integrable systems with Lax pairs spectral problems, including the soliton solution of the Korteweg-de Vries (KdV) euqation and modified KdV equation, rogue wave solution of the nonlinear Schrödinger equation, kink solution of the sine-Gordon equation, non-smooth peakon solution of the Camassa-Holm equation and pulse solution of the short pulse equation, as well as the line-soliton solution of Kadomtsev-Petviashvili equation and lump solution of high-dimensional KdV equation. The innovation of this work lies in the pioneering integration of Lax pairs informed of integrable systems into deep neural networks, thereby presenting a fresh methodology and pathway for investigating data-driven localized wave solutions and Lax pairs spectral problems.
△ Less
Submitted 10 January, 2024;
originally announced January 2024.
-
Study Duration Prediction for Clinical Trials with Time-to-Event Endpoints Using Mixture Distributions Accounting for Heterogeneous Population
Authors:
Hong Zhang,
Jie Pu,
Shibing Deng,
Satrajit Roychoudhury,
Haitao Chu,
Douglas Robinson
Abstract:
In the era of precision medicine, more and more clinical trials are now driven or guided by biomarkers, which are patient characteristics objectively measured and evaluated as indicators of normal biological processes, pathogenic processes, or pharmacologic responses to therapeutic interventions. With the overarching objective to optimize and personalize disease management, biomarker-guided clinic…
▽ More
In the era of precision medicine, more and more clinical trials are now driven or guided by biomarkers, which are patient characteristics objectively measured and evaluated as indicators of normal biological processes, pathogenic processes, or pharmacologic responses to therapeutic interventions. With the overarching objective to optimize and personalize disease management, biomarker-guided clinical trials increase the efficiency by appropriately utilizing prognostic or predictive biomarkers in the design. However, the efficiency gain is often not quantitatively compared to the traditional all-comers design, in which a faster enrollment rate is expected (e.g. due to no restriction to biomarker positive patients) potentially leading to a shorter duration. To accurately predict biomarker-guided trial duration, we propose a general framework using mixture distributions accounting for heterogeneous population. Extensive simulations are performed to evaluate the impact of heterogeneous population and the dynamics of biomarker characteristics and disease on the study duration. Several influential parameters including median survival time, enrollment rate, biomarker prevalence and effect size are identitied. Re-assessments of two publicly available trials are conducted to empirically validate the prediction accuracy and to demonstrate the practical utility. The R package \emph{detest} is developed to implement the proposed method and is publicly available on CRAN.
△ Less
Submitted 31 December, 2023;
originally announced January 2024.
-
Sampling Algorithms for Butterfly Counting on Temporal Bipartite Graphs
Authors:
Jiaxi Pu,
Yanhao Wang,
Yuchen Li,
Xuan Zhou
Abstract:
Temporal bipartite graphs are widely used to denote time-evolving relationships between two disjoint sets of nodes, such as customer-product interactions in E-commerce and user-group memberships in social networks. Temporal butterflies, $(2,2)$-bicliques that occur within a short period and in a prescribed order, are essential in modeling the structural and sequential patterns of such graphs. Coun…
▽ More
Temporal bipartite graphs are widely used to denote time-evolving relationships between two disjoint sets of nodes, such as customer-product interactions in E-commerce and user-group memberships in social networks. Temporal butterflies, $(2,2)$-bicliques that occur within a short period and in a prescribed order, are essential in modeling the structural and sequential patterns of such graphs. Counting the number of temporal butterflies is thus a fundamental task in analyzing temporal bipartite graphs. However, existing algorithms for butterfly counting on static bipartite graphs and motif counting on temporal unipartite graphs are inefficient for this purpose. In this paper, we present a general framework with three sampling strategies for temporal butterfly counting. Since exact counting can be time-consuming on large graphs, our approach alternatively computes approximate estimates accurately and efficiently. We also provide analytical bounds on the number of samples each strategy requires to obtain estimates with small relative errors and high probability. We finally evaluate our framework on six real-world datasets and demonstrate its superior accuracy and efficiency compared to several baselines. Overall, our proposed framework and sampling strategies provide efficient and accurate approaches to approximating temporal butterfly counts on large-scale temporal bipartite graphs.
△ Less
Submitted 18 October, 2023;
originally announced October 2023.
-
Multi-stage Large Language Model Correction for Speech Recognition
Authors:
Jie Pu,
Thai-Son Nguyen,
Sebastian Stüker
Abstract:
In this paper, we investigate the usage of large language models (LLMs) to improve the performance of competitive speech recognition systems. Different from previous LLM-based ASR error correction methods, we propose a novel multi-stage approach that utilizes uncertainty estimation of ASR outputs and reasoning capability of LLMs. Specifically, the proposed approach has two stages: the first stage…
▽ More
In this paper, we investigate the usage of large language models (LLMs) to improve the performance of competitive speech recognition systems. Different from previous LLM-based ASR error correction methods, we propose a novel multi-stage approach that utilizes uncertainty estimation of ASR outputs and reasoning capability of LLMs. Specifically, the proposed approach has two stages: the first stage is about ASR uncertainty estimation and exploits N-best list hypotheses to identify less reliable transcriptions; The second stage works on these identified transcriptions and performs LLM-based corrections. This correction task is formulated as a multi-step rule-based LLM reasoning process, which uses explicitly written rules in prompts to decompose the task into concrete reasoning steps. Our experimental results demonstrate the effectiveness of the proposed method by showing 10% ~ 20% relative improvement in WER over competitive ASR systems -- across multiple test domains and in zero-shot settings.
△ Less
Submitted 17 June, 2024; v1 submitted 17 October, 2023;
originally announced October 2023.
-
Revisiting Multi-modal 3D Semantic Segmentation in Real-world Autonomous Driving
Authors:
Feng Jiang,
Chaoping Tu,
Gang Zhang,
Jun Li,
Hanqing Huang,
Junyu Lin,
Di Feng,
Jian Pu
Abstract:
LiDAR and camera are two critical sensors for multi-modal 3D semantic segmentation and are supposed to be fused efficiently and robustly to promise safety in various real-world scenarios. However, existing multi-modal methods face two key challenges: 1) difficulty with efficient deployment and real-time execution; and 2) drastic performance degradation under weak calibration between LiDAR and came…
▽ More
LiDAR and camera are two critical sensors for multi-modal 3D semantic segmentation and are supposed to be fused efficiently and robustly to promise safety in various real-world scenarios. However, existing multi-modal methods face two key challenges: 1) difficulty with efficient deployment and real-time execution; and 2) drastic performance degradation under weak calibration between LiDAR and cameras. To address these challenges, we propose CPGNet-LCF, a new multi-modal fusion framework extending the LiDAR-only CPGNet. CPGNet-LCF solves the first challenge by inheriting the easy deployment and real-time capabilities of CPGNet. For the second challenge, we introduce a novel weak calibration knowledge distillation strategy during training to improve the robustness against the weak calibration. CPGNet-LCF achieves state-of-the-art performance on the nuScenes and SemanticKITTI benchmarks. Remarkably, it can be easily deployed to run in 20ms per frame on a single Tesla V100 GPU using TensorRT TF16 mode. Furthermore, we benchmark performance over four weak calibration levels, demonstrating the robustness of our proposed approach.
△ Less
Submitted 12 October, 2023;
originally announced October 2023.
-
Single-shot recognition of orbital angular momentum from speckles with spatially multiplexed point detection (SMPD)
Authors:
Zhiyuan Wang,
Haoran Li,
Qi Zhao,
Vinu R V,
Xiaocong Yuan,
Ziyang Chen,
Puxiang Lai,
Jixiong Pu
Abstract:
The widely adopted optical information detection strategies nowadays can be roughly classified into spatial-resolve-based and time-sequence-based methods. The former is highly reliant on detectors with spatial resolution that works in a limited spectrum range to capture the intensity distribution of the object, whereas the latter employs single-pixel detectors (SPDs) with improved reliability and…
▽ More
The widely adopted optical information detection strategies nowadays can be roughly classified into spatial-resolve-based and time-sequence-based methods. The former is highly reliant on detectors with spatial resolution that works in a limited spectrum range to capture the intensity distribution of the object, whereas the latter employs single-pixel detectors (SPDs) with improved reliability and stability in detection at the expense of time consumption due to sequential intensity recording. However, there is a scarcity of research on using high-performance SPDs for single-shot object information detection. To capture and extract features of objects efficiently and economically, in this work, we propose a single-shot optical information-detecting method, tentatively named spatially multiplexed point detection (SMPD) technology, in which some SPDs are randomly distributed at different locations to acquire the information of the object. To validate the validity of the proposed method, we demonstrate high-fidelity recognition of orbital angular momentum (OAM) modes from speckle patterns generated by the transmission of two orthogonally polarized vortex beams through a multimode fiber. It is feasible to apply this approach in wide applications, such as image transmission based on multiplexed-OAM decoding and recognizing hand-written digits. Compared with traditional image recognition methods, the new approach yields a recognition accuracy of over 98% with an effective detection area that is only 0.02% of its counterparts. With further engineering, the proposed method may spur many exciting developments in OAM-based optical communication systems, image classification, object detections, and other optical information detections.
△ Less
Submitted 6 October, 2023;
originally announced October 2023.
-
ADU-Depth: Attention-based Distillation with Uncertainty Modeling for Depth Estimation
Authors:
Zizhang Wu,
Zhuozheng Li,
Zhi-Gang Fan,
Yunzhe Wu,
Xiaoquan Wang,
Rui Tang,
Jian Pu
Abstract:
Monocular depth estimation is challenging due to its inherent ambiguity and ill-posed nature, yet it is quite important to many applications. While recent works achieve limited accuracy by designing increasingly complicated networks to extract features with limited spatial geometric cues from a single RGB image, we intend to introduce spatial cues by training a teacher network that leverages left-…
▽ More
Monocular depth estimation is challenging due to its inherent ambiguity and ill-posed nature, yet it is quite important to many applications. While recent works achieve limited accuracy by designing increasingly complicated networks to extract features with limited spatial geometric cues from a single RGB image, we intend to introduce spatial cues by training a teacher network that leverages left-right image pairs as inputs and transferring the learned 3D geometry-aware knowledge to the monocular student network. Specifically, we present a novel knowledge distillation framework, named ADU-Depth, with the goal of leveraging the well-trained teacher network to guide the learning of the student network, thus boosting the precise depth estimation with the help of extra spatial scene information. To enable domain adaptation and ensure effective and smooth knowledge transfer from teacher to student, we apply both attention-adapted feature distillation and focal-depth-adapted response distillation in the training stage. In addition, we explicitly model the uncertainty of depth estimation to guide distillation in both feature space and result space to better produce 3D-aware knowledge from monocular observations and thus enhance the learning for hard-to-predict image regions. Our extensive experiments on the real depth estimation datasets KITTI and DrivingStereo demonstrate the effectiveness of the proposed method, which ranked 1st on the challenging KITTI online benchmark.
△ Less
Submitted 26 September, 2023;
originally announced September 2023.
-
A Novel Approach for Effective Multi-View Clustering with Information-Theoretic Perspective
Authors:
Chenhang Cui,
Yazhou Ren,
Jingyu Pu,
Jiawei Li,
Xiaorong Pu,
Tianyi Wu,
Yutao Shi,
Lifang He
Abstract:
Multi-view clustering (MVC) is a popular technique for improving clustering performance using various data sources. However, existing methods primarily focus on acquiring consistent information while often neglecting the issue of redundancy across multiple views. This study presents a new approach called Sufficient Multi-View Clustering (SUMVC) that examines the multi-view clustering framework fro…
▽ More
Multi-view clustering (MVC) is a popular technique for improving clustering performance using various data sources. However, existing methods primarily focus on acquiring consistent information while often neglecting the issue of redundancy across multiple views. This study presents a new approach called Sufficient Multi-View Clustering (SUMVC) that examines the multi-view clustering framework from an information-theoretic standpoint. Our proposed method consists of two parts. Firstly, we develop a simple and reliable multi-view clustering method SCMVC (simple consistent multi-view clustering) that employs variational analysis to generate consistent information. Secondly, we propose a sufficient representation lower bound to enhance consistent information and minimise unnecessary information among views. The proposed SUMVC method offers a promising solution to the problem of multi-view clustering and provides a new perspective for analyzing multi-view data.
To verify the effectiveness of our model, we conducted a theoretical analysis based on the Bayes Error Rate, and experiments on multiple multi-view datasets demonstrate the superior performance of SUMVC.
△ Less
Submitted 25 September, 2023;
originally announced September 2023.
-
PointSSC: A Cooperative Vehicle-Infrastructure Point Cloud Benchmark for Semantic Scene Completion
Authors:
Yuxiang Yan,
Boda Liu,
Jianfei Ai,
Qinbu Li,
Ru Wan,
Jian Pu
Abstract:
Semantic Scene Completion (SSC) aims to jointly generate space occupancies and semantic labels for complex 3D scenes. Most existing SSC models focus on volumetric representations, which are memory-inefficient for large outdoor spaces. Point clouds provide a lightweight alternative but existing benchmarks lack outdoor point cloud scenes with semantic labels. To address this, we introduce PointSSC,…
▽ More
Semantic Scene Completion (SSC) aims to jointly generate space occupancies and semantic labels for complex 3D scenes. Most existing SSC models focus on volumetric representations, which are memory-inefficient for large outdoor spaces. Point clouds provide a lightweight alternative but existing benchmarks lack outdoor point cloud scenes with semantic labels. To address this, we introduce PointSSC, the first cooperative vehicle-infrastructure point cloud benchmark for semantic scene completion. These scenes exhibit long-range perception and minimal occlusion. We develop an automated annotation pipeline leveraging Semantic Segment Anything to efficiently assign semantics. To benchmark progress, we propose a LiDAR-based model with a Spatial-Aware Transformer for global and local feature extraction and a Completion and Segmentation Cooperative Module for joint completion and segmentation. PointSSC provides a challenging testbed to drive advances in semantic point cloud completion for real-world navigation. The code and datasets are available at https://github.com/yyxssm/PointSSC.
△ Less
Submitted 6 March, 2024; v1 submitted 22 September, 2023;
originally announced September 2023.
-
PPD: A New Valet Parking Pedestrian Fisheye Dataset for Autonomous Driving
Authors:
Zizhang Wu,
Xinyuan Chen,
Fan Song,
Yuanzhu Gan,
Tianhao Xu,
Jian Pu,
Rui Tang
Abstract:
Pedestrian detection under valet parking scenarios is fundamental for autonomous driving. However, the presence of pedestrians can be manifested in a variety of ways and postures under imperfect ambient conditions, which can adversely affect detection performance. Furthermore, models trained on publicdatasets that include pedestrians generally provide suboptimal outcomes for these valet parking sc…
▽ More
Pedestrian detection under valet parking scenarios is fundamental for autonomous driving. However, the presence of pedestrians can be manifested in a variety of ways and postures under imperfect ambient conditions, which can adversely affect detection performance. Furthermore, models trained on publicdatasets that include pedestrians generally provide suboptimal outcomes for these valet parking scenarios. In this paper, wepresent the Parking Pedestrian Dataset (PPD), a large-scale fisheye dataset to support research dealing with real-world pedestrians, especially with occlusions and diverse postures. PPD consists of several distinctive types of pedestrians captured with fisheye cameras. Additionally, we present a pedestrian detection baseline on PPD dataset, and introduce two data augmentation techniques to improve the baseline by enhancing the diversity ofthe original dataset. Extensive experiments validate the effectiveness of our novel data augmentation approaches over baselinesand the dataset's exceptional generalizability.
△ Less
Submitted 24 September, 2023; v1 submitted 19 September, 2023;
originally announced September 2023.
-
LineMarkNet: Line Landmark Detection for Valet Parking
Authors:
Zizhang Wu,
Yuanzhu Gan,
Tianhao Xu,
Rui Tang,
Jian Pu
Abstract:
We aim for accurate and efficient line landmark detection for valet parking, which is a long-standing yet unsolved problem in autonomous driving. To this end, we present a deep line landmark detection system where we carefully design the modules to be lightweight. Specifically, we first empirically design four general line landmarks including three physical lines and one novel mental line. The fou…
▽ More
We aim for accurate and efficient line landmark detection for valet parking, which is a long-standing yet unsolved problem in autonomous driving. To this end, we present a deep line landmark detection system where we carefully design the modules to be lightweight. Specifically, we first empirically design four general line landmarks including three physical lines and one novel mental line. The four line landmarks are effective for valet parking. We then develop a deep network (LineMarkNet) to detect line landmarks from surround-view cameras where we, via the pre-calibrated homography, fuse context from four separate cameras into the unified bird-eye-view (BEV) space, specifically we fuse the surroundview features and BEV features, then employ the multi-task decoder to detect multiple line landmarks where we apply the center-based strategy for object detection task, and design our graph transformer to enhance the vision transformer with hierarchical level graph reasoning for semantic segmentation task. At last, we further parameterize the detected line landmarks (e.g., intercept-slope form) whereby a novel filtering backend incorporates temporal and multi-view consistency to achieve smooth and stable detection. Moreover, we annotate a large-scale dataset to validate our method. Experimental results show that our framework achieves the enhanced performance compared with several line detection methods and validate the multi-task network's efficiency about the real-time line landmark detection on the Qualcomm 820A platform while meantime keeps superior accuracy, with our deep line landmark detection system.
△ Less
Submitted 24 September, 2023; v1 submitted 19 September, 2023;
originally announced September 2023.
-
Examining the Effect of Pre-training on Time Series Classification
Authors:
Jiashu Pu,
Shiwei Zhao,
Ling Cheng,
Yongzhu Chang,
Runze Wu,
Tangjie Lv,
Rongsheng Zhang
Abstract:
Although the pre-training followed by fine-tuning paradigm is used extensively in many fields, there is still some controversy surrounding the impact of pre-training on the fine-tuning process. Currently, experimental findings based on text and image data lack consensus. To delve deeper into the unsupervised pre-training followed by fine-tuning paradigm, we have extended previous research to a new…
▽ More
Although the pre-training followed by fine-tuning paradigm is used extensively in many fields, there is still some controversy surrounding the impact of pre-training on the fine-tuning process. Currently, experimental findings based on text and image data lack consensus. To delve deeper into the unsupervised pre-training followed by fine-tuning paradigm, we have extended previous research to a new modality: time series. In this study, we conducted a thorough examination of 150 classification datasets derived from the Univariate Time Series (UTS) and Multivariate Time Series (MTS) benchmarks. Our analysis reveals several key conclusions. (i) Pre-training can only help improve the optimization process for models that fit the data poorly, rather than those that fit the data well. (ii) Pre-training does not exhibit the effect of regularization when given sufficient training time. (iii) Pre-training can only speed up convergence if the model has sufficient ability to fit the data. (iv) Adding more pre-training data does not improve generalization, but it can strengthen the advantage of pre-training on the original data volume, such as faster convergence. (v) While both the pre-training task and the model structure determine the effectiveness of the paradigm on a given dataset, the model structure plays a more significant role.
△ Less
Submitted 11 September, 2023;
originally announced September 2023.
-
Lifted Algorithms for Symmetric Weighted First-Order Model Sampling
Authors:
Yuanhong Wang,
Juhua Pu,
Yuyi Wang,
Ondřej Kuželka
Abstract:
Weighted model counting (WMC) is the task of computing the weighted sum of all satisfying assignments (i.e., models) of a propositional formula. Similarly, weighted model sampling (WMS) aims to randomly generate models with probability proportional to their respective weights. Both WMC and WMS are hard to solve exactly, falling under the $\#\mathsf{P}$-hard complexity class. However, it is known t…
▽ More
Weighted model counting (WMC) is the task of computing the weighted sum of all satisfying assignments (i.e., models) of a propositional formula. Similarly, weighted model sampling (WMS) aims to randomly generate models with probability proportional to their respective weights. Both WMC and WMS are hard to solve exactly, falling under the $\#\mathsf{P}$-hard complexity class. However, it is known that the counting problem may sometimes be tractable, if the propositional formula can be compactly represented and expressed in first-order logic. In such cases, model counting problems can be solved in time polynomial in the domain size, and are known as domain-liftable. The following question then arises: Is it also the case for weighted model sampling? This paper addresses this question and answers it affirmatively. Specifically, we prove the domain-liftability under sampling for the two-variables fragment of first-order logic with counting quantifiers in this paper, by devising an efficient sampling algorithm for this fragment that runs in time polynomial in the domain size. We then further show that this result continues to hold even in the presence of cardinality constraints. To empirically verify our approach, we conduct experiments over various first-order formulas designed for the uniform generation of combinatorial structures and sampling in statistical-relational models. The results demonstrate that our algorithm outperforms a start-of-the-art WMS sampler by a substantial margin, confirming the theoretical results.
△ Less
Submitted 14 June, 2024; v1 submitted 17 August, 2023;
originally announced August 2023.
-
Sudowoodo: a Chinese Lyric Imitation System with Source Lyrics
Authors:
Yongzhu Chang,
Rongsheng Zhang,
Lin Jiang,
Qihang Chen,
Le Zhang,
Jiashu Pu
Abstract:
Lyrics generation is a well-known application in natural language generation research, with several previous studies focusing on generating accurate lyrics using precise control such as keywords, rhymes, etc. However, lyrics imitation, which involves writing new lyrics by imitating the style and content of the source lyrics, remains a challenging task due to the lack of a parallel corpus. In this…
▽ More
Lyrics generation is a well-known application in natural language generation research, with several previous studies focusing on generating accurate lyrics using precise control such as keywords, rhymes, etc. However, lyrics imitation, which involves writing new lyrics by imitating the style and content of the source lyrics, remains a challenging task due to the lack of a parallel corpus. In this paper, we introduce \textbf{\textit{Sudowoodo}}, a Chinese lyrics imitation system that can generate new lyrics based on the text of source lyrics. To address the issue of lacking a parallel training corpus for lyrics imitation, we propose a novel framework to construct a parallel corpus based on a keyword-based lyrics model from source lyrics. Then the pairs \textit{(new lyrics, source lyrics)} are used to train the lyrics imitation model. During the inference process, we utilize a post-processing module to filter and rank the generated lyrics, selecting the highest-quality ones. We incorporated audio information and aligned the lyrics with the audio to form the songs as a bonus. The human evaluation results show that our framework can perform better lyric imitation. Meanwhile, the \textit{Sudowoodo} system and demo video of the system is available at \href{https://Sudowoodo.apps-hp.danlu.netease.com/}{Sudowoodo} and \href{https://youtu.be/u5BBT_j1L5M}{https://youtu.be/u5BBT\_j1L5M}.
△ Less
Submitted 8 August, 2023;
originally announced August 2023.
-
I-WAS: a Data Augmentation Method with GPT-2 for Simile Detection
Authors:
Yongzhu Chang,
Rongsheng Zhang,
Jiashu Pu
Abstract:
Simile detection is a valuable task for many natural language processing (NLP)-based applications, particularly in the field of literature. However, existing research on simile detection often relies on corpora that are limited in size and do not adequately represent the full range of simile forms. To address this issue, we propose a simile data augmentation method based on \textbf{W}ord replaceme…
▽ More
Simile detection is a valuable task for many natural language processing (NLP)-based applications, particularly in the field of literature. However, existing research on simile detection often relies on corpora that are limited in size and do not adequately represent the full range of simile forms. To address this issue, we propose a simile data augmentation method based on \textbf{W}ord replacement And Sentence completion using the GPT-2 language model. Our iterative process called I-WAS, is designed to improve the quality of the augmented sentences. To better evaluate the performance of our method in real-world applications, we have compiled a corpus containing a more diverse set of simile forms for experimentation. Our experimental results demonstrate the effectiveness of our proposed data augmentation method for simile detection.
△ Less
Submitted 8 August, 2023;
originally announced August 2023.
-
Understanding Depth Map Progressively: Adaptive Distance Interval Separation for Monocular 3d Object Detection
Authors:
Xianhui Cheng,
Shoumeng Qiu,
Zhikang Zou,
Jian Pu,
Xiangyang Xue
Abstract:
Monocular 3D object detection aims to locate objects in different scenes with just a single image. Due to the absence of depth information, several monocular 3D detection techniques have emerged that rely on auxiliary depth maps from the depth estimation task. There are multiple approaches to understanding the representation of depth maps, including treating them as pseudo-LiDAR point clouds, leve…
▽ More
Monocular 3D object detection aims to locate objects in different scenes with just a single image. Due to the absence of depth information, several monocular 3D detection techniques have emerged that rely on auxiliary depth maps from the depth estimation task. There are multiple approaches to understanding the representation of depth maps, including treating them as pseudo-LiDAR point clouds, leveraging implicit end-to-end learning of depth information, or considering them as an image input. However, these methods have certain drawbacks, such as their reliance on the accuracy of estimated depth maps and suboptimal utilization of depth maps due to their image-based nature. While LiDAR-based methods and convolutional neural networks (CNNs) can be utilized for pseudo point clouds and depth maps, respectively, it is always an alternative. In this paper, we propose a framework named the Adaptive Distance Interval Separation Network (ADISN) that adopts a novel perspective on understanding depth maps, as a form that lies between LiDAR and images. We utilize an adaptive separation approach that partitions the depth map into various subgraphs based on distance and treats each of these subgraphs as an individual image for feature extraction. After adaptive separations, each subgraph solely contains pixels within a learned interval range. If there is a truncated object within this range, an evident curved edge will appear, which we can leverage for texture extraction using CNNs to obtain rich depth information in pixels. Meanwhile, to mitigate the inaccuracy of depth estimation, we designed an uncertainty module. To take advantage of both images and depth maps, we use different branches to learn localization detection tasks and appearance tasks separately.
△ Less
Submitted 19 June, 2023;
originally announced June 2023.
-
Neighborhood-based Hard Negative Mining for Sequential Recommendation
Authors:
Lu Fan,
Jiashu Pu,
Rongsheng Zhang,
Xiao-Ming Wu
Abstract:
Negative sampling plays a crucial role in training successful sequential recommendation models. Instead of merely employing random negative sample selection, numerous strategies have been proposed to mine informative negative samples to enhance training and performance. However, few of these approaches utilize structural information. In this work, we observe that as training progresses, the distri…
▽ More
Negative sampling plays a crucial role in training successful sequential recommendation models. Instead of merely employing random negative sample selection, numerous strategies have been proposed to mine informative negative samples to enhance training and performance. However, few of these approaches utilize structural information. In this work, we observe that as training progresses, the distributions of node-pair similarities in different groups with varying degrees of neighborhood overlap change significantly, suggesting that item pairs in distinct groups may possess different negative relationships. Motivated by this observation, we propose a Graph-based Negative sampling approach based on Neighborhood Overlap (GNNO) to exploit structural information hidden in user behaviors for negative mining. GNNO first constructs a global weighted item transition graph using training sequences. Subsequently, it mines hard negative samples based on the degree of overlap with the target item on the graph. Furthermore, GNNO employs curriculum learning to control the hardness of negative samples, progressing from easy to difficult. Extensive experiments on three Amazon benchmarks demonstrate GNNO's effectiveness in consistently enhancing the performance of various state-of-the-art models and surpassing existing negative sampling strategies. The code will be released at \url{https://github.com/floatSDSDS/GNNO}.
△ Less
Submitted 12 June, 2023;
originally announced June 2023.
-
Retrospective: EIE: Efficient Inference Engine on Sparse and Compressed Neural Network
Authors:
Song Han,
Xingyu Liu,
Huizi Mao,
Jing Pu,
Ardavan Pedram,
Mark A. Horowitz,
William J. Dally
Abstract:
EIE proposed to accelerate pruned and compressed neural networks, exploiting weight sparsity, activation sparsity, and 4-bit weight-sharing in neural network accelerators. Since published in ISCA'16, it opened a new design space to accelerate pruned and sparse neural networks and spawned many algorithm-hardware co-designs for model compression and acceleration, both in academia and commercial AI c…
▽ More
EIE proposed to accelerate pruned and compressed neural networks, exploiting weight sparsity, activation sparsity, and 4-bit weight-sharing in neural network accelerators. Since published in ISCA'16, it opened a new design space to accelerate pruned and sparse neural networks and spawned many algorithm-hardware co-designs for model compression and acceleration, both in academia and commercial AI chips. In retrospect, we review the background of this project, summarize the pros and cons, and discuss new opportunities where pruning, sparsity, and low precision can accelerate emerging deep learning workloads.
△ Less
Submitted 15 June, 2023;
originally announced June 2023.
-
Learning Monocular Depth in Dynamic Environment via Context-aware Temporal Attention
Authors:
Zizhang Wu,
Zhuozheng Li,
Zhi-Gang Fan,
Yunzhe Wu,
Yuanzhu Gan,
Jian Pu,
Xianzhi Li
Abstract:
The monocular depth estimation task has recently revealed encouraging prospects, especially for the autonomous driving task. To tackle the ill-posed problem of 3D geometric reasoning from 2D monocular images, multi-frame monocular methods are developed to leverage the perspective correlation information from sequential temporal frames. However, moving objects such as cars and trains usually violat…
▽ More
The monocular depth estimation task has recently revealed encouraging prospects, especially for the autonomous driving task. To tackle the ill-posed problem of 3D geometric reasoning from 2D monocular images, multi-frame monocular methods are developed to leverage the perspective correlation information from sequential temporal frames. However, moving objects such as cars and trains usually violate the static scene assumption, leading to feature inconsistency deviation and misaligned cost values, which would mislead the optimization algorithm. In this work, we present CTA-Depth, a Context-aware Temporal Attention guided network for multi-frame monocular Depth estimation. Specifically, we first apply a multi-level attention enhancement module to integrate multi-level image features to obtain an initial depth and pose estimation. Then the proposed CTA-Refiner is adopted to alternatively optimize the depth and pose. During the refinement process, context-aware temporal attention (CTA) is developed to capture the global temporal-context correlations to maintain the feature consistency and estimation integrity of moving objects. In particular, we propose a long-range geometry embedding (LGE) module to produce a long-range temporal geometry prior. Our approach achieves significant improvements over state-of-the-art approaches on three benchmark datasets.
△ Less
Submitted 12 May, 2023;
originally announced May 2023.
-
Deep Multi-View Subspace Clustering with Anchor Graph
Authors:
Chenhang Cui,
Yazhou Ren,
Jingyu Pu,
Xiaorong Pu,
Lifang He
Abstract:
Deep multi-view subspace clustering (DMVSC) has recently attracted increasing attention due to its promising performance. However, existing DMVSC methods still have two issues: (1) they mainly focus on using autoencoders to nonlinearly embed the data, while the embedding may be suboptimal for clustering because the clustering objective is rarely considered in autoencoders, and (2) existing methods…
▽ More
Deep multi-view subspace clustering (DMVSC) has recently attracted increasing attention due to its promising performance. However, existing DMVSC methods still have two issues: (1) they mainly focus on using autoencoders to nonlinearly embed the data, while the embedding may be suboptimal for clustering because the clustering objective is rarely considered in autoencoders, and (2) existing methods typically have a quadratic or even cubic complexity, which makes it challenging to deal with large-scale data. To address these issues, in this paper we propose a novel deep multi-view subspace clustering method with anchor graph (DMCAG). To be specific, DMCAG firstly learns the embedded features for each view independently, which are used to obtain the subspace representations. To significantly reduce the complexity, we construct an anchor graph with small size for each view. Then, spectral clustering is performed on an integrated anchor graph to obtain pseudo-labels. To overcome the negative impact caused by suboptimal embedded features, we use pseudo-labels to refine the embedding process to make it more suitable for the clustering task. Pseudo-labels and embedded features are updated alternately. Furthermore, we design a strategy to keep the consistency of the labels based on contrastive learning to enhance the clustering performance. Empirical studies on real-world datasets show that our method achieves superior clustering performance over other state-of-the-art methods.
△ Less
Submitted 11 May, 2023;
originally announced May 2023.
-
A Multi-modal Garden Dataset and Hybrid 3D Dense Reconstruction Framework Based on Panoramic Stereo Images for a Trimming Robot
Authors:
Can Pu,
Chuanyu Yang,
Jinnian Pu,
Radim Tylecek,
Robert B. Fisher
Abstract:
Recovering an outdoor environment's surface mesh is vital for an agricultural robot during task planning and remote visualization. Our proposed solution is based on a newly-designed panoramic stereo camera along with a hybrid novel software framework that consists of three fusion modules. The panoramic stereo camera with a pentagon shape consists of 5 stereo vision camera pairs to stream synchroni…
▽ More
Recovering an outdoor environment's surface mesh is vital for an agricultural robot during task planning and remote visualization. Our proposed solution is based on a newly-designed panoramic stereo camera along with a hybrid novel software framework that consists of three fusion modules. The panoramic stereo camera with a pentagon shape consists of 5 stereo vision camera pairs to stream synchronized panoramic stereo images for the following three fusion modules. In the disparity fusion module, rectified stereo images produce the initial disparity maps using multiple stereo vision algorithms. Then, these initial disparity maps, along with the intensity images, are input into a disparity fusion network to produce refined disparity maps. Next, the refined disparity maps are converted into full-view point clouds or single-view point clouds for the pose fusion module. The pose fusion module adopts a two-stage global-coarse-to-local-fine strategy. In the first stage, each pair of full-view point clouds is registered by a global point cloud matching algorithm to estimate the transformation for a global pose graph's edge, which effectively implements loop closure. In the second stage, a local point cloud matching algorithm is used to match single-view point clouds in different nodes. Next, we locally refine the poses of all corresponding edges in the global pose graph using three proposed rules, thus constructing a refined pose graph. The refined pose graph is optimized to produce a global pose trajectory for volumetric fusion. In the volumetric fusion module, the global poses of all the nodes are used to integrate the single-view point clouds into the volume to produce the mesh of the whole garden. The proposed framework and its three fusion modules are tested on a real outdoor garden dataset to show the superiority of the performance.
△ Less
Submitted 10 May, 2023;
originally announced May 2023.
-
Complex dynamics on the one-dimensional quantum droplets via time piecewise PINNs
Authors:
Juncai Pu,
Yong Chen
Abstract:
The dynamics of one-dimensional quantum droplets and the landing applications of deep learning are recent research hotspots. In this work, we propose a novel time piecewise physics-informed neural networks (PINNs) to study complex dynamics on the one-dimensional quantum droplets by solving the corresponding amended Gross-Pitaevskii equation. The training effect of this network model in the long ti…
▽ More
The dynamics of one-dimensional quantum droplets and the landing applications of deep learning are recent research hotspots. In this work, we propose a novel time piecewise physics-informed neural networks (PINNs) to study complex dynamics on the one-dimensional quantum droplets by solving the corresponding amended Gross-Pitaevskii equation. The training effect of this network model in the long time domain is far better than that of the conventional PINNs, and each of its subnetworks is independent and highly adjustable. By using time piecewise PINNs with scarce training points, we not only study intrinsic modulation of single droplet and collision between two droplets, but also excite the breathers on droplet background. Intriguingly, we obtain an interference pattern from training result of collision between two droplets, which is a significant feature of the interplay of coherent matter waves. The numerical results showcase that different parameters may lead to completely different dynamic behaviors under the same initial condition in a nonlinear non-integrable system. Our results provide the significant guidance for intrinsic modulation of single droplet, droplet collision and breathers excitation via deep learning technology.
△ Less
Submitted 2 May, 2023; v1 submitted 26 February, 2023;
originally announced May 2023.
-
Multi-to-Single Knowledge Distillation for Point Cloud Semantic Segmentation
Authors:
Shoumeng Qiu,
Feng Jiang,
Haiqiang Zhang,
Xiangyang Xue,
Jian Pu
Abstract:
3D point cloud semantic segmentation is one of the fundamental tasks for environmental understanding. Although significant progress has been made in recent years, the performance of classes with few examples or few points is still far from satisfactory. In this paper, we propose a novel multi-to-single knowledge distillation framework for the 3D point cloud semantic segmentation task to boost the…
▽ More
3D point cloud semantic segmentation is one of the fundamental tasks for environmental understanding. Although significant progress has been made in recent years, the performance of classes with few examples or few points is still far from satisfactory. In this paper, we propose a novel multi-to-single knowledge distillation framework for the 3D point cloud semantic segmentation task to boost the performance of those hard classes. Instead of fusing all the points of multi-scans directly, only the instances that belong to the previously defined hard classes are fused. To effectively and sufficiently distill valuable knowledge from multi-scans, we leverage a multilevel distillation framework, i.e., feature representation distillation, logit distillation, and affinity distillation. We further develop a novel instance-aware affinity distillation algorithm for capturing high-level structural knowledge to enhance the distillation efficacy for hard classes. Finally, we conduct experiments on the SemanticKITTI dataset, and the results on both the validation and test sets demonstrate that our method yields substantial improvements compared with the baseline method. The code is available at \Url{https://github.com/skyshoumeng/M2SKD}.
△ Less
Submitted 28 April, 2023;
originally announced April 2023.
-
Knowledge Distillation from 3D to Bird's-Eye-View for LiDAR Semantic Segmentation
Authors:
Feng Jiang,
Heng Gao,
Shoumeng Qiu,
Haiqiang Zhang,
Ru Wan,
Jian Pu
Abstract:
LiDAR point cloud segmentation is one of the most fundamental tasks for autonomous driving scene understanding. However, it is difficult for existing models to achieve both high inference speed and accuracy simultaneously. For example, voxel-based methods perform well in accuracy, while Bird's-Eye-View (BEV)-based methods can achieve real-time inference. To overcome this issue, we develop an effec…
▽ More
LiDAR point cloud segmentation is one of the most fundamental tasks for autonomous driving scene understanding. However, it is difficult for existing models to achieve both high inference speed and accuracy simultaneously. For example, voxel-based methods perform well in accuracy, while Bird's-Eye-View (BEV)-based methods can achieve real-time inference. To overcome this issue, we develop an effective 3D-to-BEV knowledge distillation method that transfers rich knowledge from 3D voxel-based models to BEV-based models. Our framework mainly consists of two modules: the voxel-to-pillar distillation module and the label-weight distillation module. Voxel-to-pillar distillation distills sparse 3D features to BEV features for middle layers to make the BEV-based model aware of more structural and geometric information. Label-weight distillation helps the model pay more attention to regions with more height information. Finally, we conduct experiments on the SemanticKITTI dataset and Paris-Lille-3D. The results on SemanticKITTI show more than 5% improvement on the test set, especially for classes such as motorcycle and person, with more than 15% improvement. The code can be accessed at https://github.com/fengjiang5/Knowledge-Distillation-from-Cylinder3D-to-PolarNet.
△ Less
Submitted 22 April, 2023;
originally announced April 2023.
-
Network Pruning Spaces
Authors:
Xuanyu He,
Yu-I Yang,
Ran Song,
Jiachen Pu,
Conggang Hu,
Feijun Jiang,
Wei Zhang,
Huanghao Ding
Abstract:
Network pruning techniques, including weight pruning and filter pruning, reveal that most state-of-the-art neural networks can be accelerated without a significant performance drop. This work focuses on filter pruning which enables accelerated inference with any off-the-shelf deep learning library and hardware. We propose the concept of \emph{network pruning spaces} that parametrize populations of…
▽ More
Network pruning techniques, including weight pruning and filter pruning, reveal that most state-of-the-art neural networks can be accelerated without a significant performance drop. This work focuses on filter pruning which enables accelerated inference with any off-the-shelf deep learning library and hardware. We propose the concept of \emph{network pruning spaces} that parametrize populations of subnetwork architectures. Based on this concept, we explore the structure aspect of subnetworks that result in minimal loss of accuracy in different pruning regimes and arrive at a series of observations by comparing subnetwork distributions. We conjecture through empirical studies that there exists an optimal FLOPs-to-parameter-bucket ratio related to the design of original network in a pruning regime. Statistically, the structure of a winning subnetwork guarantees an approximately optimal ratio in this regime. Upon our conjectures, we further refine the initial pruning space to reduce the cost of searching a good subnetwork architecture. Our experimental results on ImageNet show that the subnetwork we found is superior to those from the state-of-the-art pruning methods under comparable FLOPs.
△ Less
Submitted 19 April, 2023;
originally announced April 2023.
-
SUPS: A Simulated Underground Parking Scenario Dataset for Autonomous Driving
Authors:
Jiawei Hou,
Qi Chen,
Yurong Cheng,
Guang Chen,
Xiangyang Xue,
Taiping Zeng,
Jian Pu
Abstract:
Automatic underground parking has attracted considerable attention as the scope of autonomous driving expands. The auto-vehicle is supposed to obtain the environmental information, track its location, and build a reliable map of the scenario. Mainstream solutions consist of well-trained neural networks and simultaneous localization and mapping (SLAM) methods, which need numerous carefully labeled…
▽ More
Automatic underground parking has attracted considerable attention as the scope of autonomous driving expands. The auto-vehicle is supposed to obtain the environmental information, track its location, and build a reliable map of the scenario. Mainstream solutions consist of well-trained neural networks and simultaneous localization and mapping (SLAM) methods, which need numerous carefully labeled images and multiple sensor estimations. However, there is a lack of underground parking scenario datasets with multiple sensors and well-labeled images that support both SLAM tasks and perception tasks, such as semantic segmentation and parking slot detection. In this paper, we present SUPS, a simulated dataset for underground automatic parking, which supports multiple tasks with multiple sensors and multiple semantic labels aligned with successive images according to timestamps. We intend to cover the defect of existing datasets with the variability of environments and the diversity and accessibility of sensors in the virtual scene. Specifically, the dataset records frames from four surrounding fisheye cameras, two forward pinhole cameras, a depth camera, and data from LiDAR, inertial measurement unit (IMU), GNSS. Pixel-level semantic labels are provided for objects, especially ground signs such as arrows, parking lines, lanes, and speed bumps. Perception, 3D reconstruction, depth estimation, and SLAM, and other relative tasks are supported by our dataset. We also evaluate the state-of-the-art SLAM algorithms and perception models on our dataset. Finally, we open source our virtual 3D scene built based on Unity Engine and release our dataset at https://github.com/jarvishou829/SUPS.
△ Less
Submitted 24 February, 2023;
originally announced February 2023.
-
MonoPGC: Monocular 3D Object Detection with Pixel Geometry Contexts
Authors:
Zizhang Wu,
Yuanzhu Gan,
Lei Wang,
Guilian Chen,
Jian Pu
Abstract:
Monocular 3D object detection reveals an economical but challenging task in autonomous driving. Recently center-based monocular methods have developed rapidly with a great trade-off between speed and accuracy, where they usually depend on the object center's depth estimation via 2D features. However, the visual semantic features without sufficient pixel geometry information, may affect the perform…
▽ More
Monocular 3D object detection reveals an economical but challenging task in autonomous driving. Recently center-based monocular methods have developed rapidly with a great trade-off between speed and accuracy, where they usually depend on the object center's depth estimation via 2D features. However, the visual semantic features without sufficient pixel geometry information, may affect the performance of clues for spatial 3D detection tasks. To alleviate this, we propose MonoPGC, a novel end-to-end Monocular 3D object detection framework with rich Pixel Geometry Contexts. We introduce the pixel depth estimation as our auxiliary task and design depth cross-attention pyramid module (DCPM) to inject local and global depth geometry knowledge into visual features. In addition, we present the depth-space-aware transformer (DSAT) to integrate 3D space position and depth-aware features efficiently. Besides, we design a novel depth-gradient positional encoding (DGPE) to bring more distinct pixel geometry contexts into the transformer for better object detection. Extensive experiments demonstrate that our method achieves the state-of-the-art performance on the KITTI dataset.
△ Less
Submitted 21 February, 2023;
originally announced February 2023.
-
MVFusion: Multi-View 3D Object Detection with Semantic-aligned Radar and Camera Fusion
Authors:
Zizhang Wu,
Guilian Chen,
Yuanzhu Gan,
Lei Wang,
Jian Pu
Abstract:
Multi-view radar-camera fused 3D object detection provides a farther detection range and more helpful features for autonomous driving, especially under adverse weather. The current radar-camera fusion methods deliver kinds of designs to fuse radar information with camera data. However, these fusion approaches usually adopt the straightforward concatenation operation between multi-modal features, w…
▽ More
Multi-view radar-camera fused 3D object detection provides a farther detection range and more helpful features for autonomous driving, especially under adverse weather. The current radar-camera fusion methods deliver kinds of designs to fuse radar information with camera data. However, these fusion approaches usually adopt the straightforward concatenation operation between multi-modal features, which ignores the semantic alignment with radar features and sufficient correlations across modals. In this paper, we present MVFusion, a novel Multi-View radar-camera Fusion method to achieve semantic-aligned radar features and enhance the cross-modal information interaction. To achieve so, we inject the semantic alignment into the radar features via the semantic-aligned radar encoder (SARE) to produce image-guided radar features. Then, we propose the radar-guided fusion transformer (RGFT) to fuse our radar and image features to strengthen the two modals' correlation from the global scope via the cross-attention mechanism. Extensive experiments show that MVFusion achieves state-of-the-art performance (51.7% NDS and 45.3% mAP) on the nuScenes dataset. We shall release our code and trained networks upon publication.
△ Less
Submitted 21 February, 2023;
originally announced February 2023.
-
A General Mobile Manipulator Automation Framework for Flexible Manufacturing in Hostile Industrial Environments
Authors:
Can Pu,
Chuanyu Yang,
Jinnian Pu,
Robert B. Fisher
Abstract:
To enable a mobile manipulator to perform human tasks from a single teaching demonstration is vital to flexible manufacturing. We call our proposed method MMPA (Mobile Manipulator Process Automation with One-shot Teaching). Currently, there is no effective and robust MMPA framework which is not influenced by harsh industrial environments and the mobile base's parking precision. The proposed MMPA f…
▽ More
To enable a mobile manipulator to perform human tasks from a single teaching demonstration is vital to flexible manufacturing. We call our proposed method MMPA (Mobile Manipulator Process Automation with One-shot Teaching). Currently, there is no effective and robust MMPA framework which is not influenced by harsh industrial environments and the mobile base's parking precision. The proposed MMPA framework consists of two stages: collecting data (mobile base's location, environment information, end-effector's path) in the teaching stage for robot learning; letting the end-effector repeat the nearly same path as the reference path in the world frame to reproduce the work in the automation stage. More specifically, in the automation stage, the robot navigates to the specified location without the need of a precise parking. Then, based on colored point cloud registration, the proposed IPE (Iterative Pose Estimation by Eye & Hand) algorithm could estimate the accurate 6D relative parking pose of the robot arm base without the need of any marker. Finally, the robot could learn the error compensation from the parking pose's bias to modify the end-effector's path to make it repeat a nearly same path in the world coordinate system as recorded in the teaching stage. Hundreds of trials have been conducted with a real mobile manipulator to show the superior robustness of the system and the accuracy of the process automation regardless of the harsh industrial conditions and parking precision. For the released code, please contact marketing@amigaga.com
△ Less
Submitted 9 February, 2023;
originally announced February 2023.
-
Evidence of high-temperature exciton condensation in a two-dimensional semimetal
Authors:
Qiang Gao,
Yang-hao Chan,
Yuzhe Wang,
Haotian Zhang,
Jinxu Pu,
Shengtao Cui,
Yichen Yang,
Zhengtai Liu,
Dawei Shen,
Zhe Sun,
Juan Jiang,
Tai C. Chiang,
Peng Chen
Abstract:
Electrons and holes can spontaneously form excitons and condense in a semimetal or semiconductor, as predicted decades ago. This type of Bose condensation can happen at much higher temperatures in comparison with dilute atomic gases. Two-dimensional (2D) materials with reduced Coulomb screening around the Fermi level are promising for realizing such a system. Here we report a change in the band st…
▽ More
Electrons and holes can spontaneously form excitons and condense in a semimetal or semiconductor, as predicted decades ago. This type of Bose condensation can happen at much higher temperatures in comparison with dilute atomic gases. Two-dimensional (2D) materials with reduced Coulomb screening around the Fermi level are promising for realizing such a system. Here we report a change in the band structure accompanied by a phase transition at about 180 K in single-layer ZrTe2 based on angle-resolved photoemission spectroscopy (ARPES) measurements. Below the transition temperature, gap opening and development of an ultra-flat band top around the zone center are observed. This gap and the phase transition are rapidly suppressed with extra carrier densities introduced by adding more layers or dopants on the surface. The results suggest the formation of an excitonic insulating ground state in single-layer ZrTe2, and the findings are rationalized by first principles calculations and a self-consistent mean-field theory. Our study provides evidence for exciton condensation in a 2D semimetal and demonstrates strong dimensionality effects on the formation of intrinsic bound electron-hole pairs in solids.
△ Less
Submitted 7 February, 2023;
originally announced February 2023.
-
Holographic Einstein rings of a Gauss-Bonnet AdS black hole
Authors:
Xiao-Xiong Zeng,
Ke-Jian He,
Jin Pu,
Guo-ping Li
Abstract:
Based on the AdS/CFT correspondence, we studied the holographic Einstein images of a Gauss-Bonnet AdS black hole in the bulk from a given response function on AdS boundary. For the absolute amplitude of total response function, it shows that there always exists the interference pattern as the scalar wave passed through the black hole. And, the absolute amplitude depends closely on the properties o…
▽ More
Based on the AdS/CFT correspondence, we studied the holographic Einstein images of a Gauss-Bonnet AdS black hole in the bulk from a given response function on AdS boundary. For the absolute amplitude of total response function, it shows that there always exists the interference pattern as the scalar wave passed through the black hole. And, the absolute amplitude depends closely on the properties of Gaussian source and spacetime geometry. More importantly, we also find that the holographic images always appears as a ring with the concentric stripe surrounded when the observer located at the north pole. At other positions, this ring will change into a luminosity-deformed ring, or two light points. In addition, the effects of Gauss-Bonnet parameter $α$, wave source and optical system on the holographic images have been carefully addressed throughout of paper. Finally, we conclude that the holographic images in different types of black holes with some different features may shed some deep insight on the existence of a gravity dual for a given material.
△ Less
Submitted 24 June, 2023; v1 submitted 7 February, 2023;
originally announced February 2023.
-
On Exact Sampling in the Two-Variable Fragment of First-Order Logic
Authors:
Yuanhong Wang,
Juhua Pu,
Yuyi Wang,
Ondřej Kuželka
Abstract:
In this paper, we study the sampling problem for first-order logic proposed recently by Wang et al. -- how to efficiently sample a model of a given first-order sentence on a finite domain? We extend their result for the universally-quantified subfragment of two-variable logic $\mathbf{FO}^2$ ($\mathbf{UFO}^2$) to the entire fragment of $\mathbf{FO}^2$. Specifically, we prove the domain-liftability…
▽ More
In this paper, we study the sampling problem for first-order logic proposed recently by Wang et al. -- how to efficiently sample a model of a given first-order sentence on a finite domain? We extend their result for the universally-quantified subfragment of two-variable logic $\mathbf{FO}^2$ ($\mathbf{UFO}^2$) to the entire fragment of $\mathbf{FO}^2$. Specifically, we prove the domain-liftability under sampling of $\mathbf{FO}^2$, meaning that there exists a sampling algorithm for $\mathbf{FO}^2$ that runs in time polynomial in the domain size. We then further show that this result continues to hold even in the presence of counting constraints, such as $\forall x\exists_{=k} y: \varphi(x,y)$ and $\exists_{=k} x\forall y: \varphi(x,y)$, for some quantifier-free formula $\varphi(x,y)$. Our proposed method is constructive, and the resulting sampling algorithms have potential applications in various areas, including the uniform generation of combinatorial structures and sampling in statistical-relational models such as Markov logic networks and probabilistic logic programs.
△ Less
Submitted 6 May, 2023; v1 submitted 6 February, 2023;
originally announced February 2023.
-
Attention-Based Depth Distillation with 3D-Aware Positional Encoding for Monocular 3D Object Detection
Authors:
Zizhang Wu,
Yunzhe Wu,
Jian Pu,
Xianzhi Li,
Xiaoquan Wang
Abstract:
Monocular 3D object detection is a low-cost but challenging task, as it requires generating accurate 3D localization solely from a single image input. Recent developed depth-assisted methods show promising results by using explicit depth maps as intermediate features, which are either precomputed by monocular depth estimation networks or jointly evaluated with 3D object detection. However, inevita…
▽ More
Monocular 3D object detection is a low-cost but challenging task, as it requires generating accurate 3D localization solely from a single image input. Recent developed depth-assisted methods show promising results by using explicit depth maps as intermediate features, which are either precomputed by monocular depth estimation networks or jointly evaluated with 3D object detection. However, inevitable errors from estimated depth priors may lead to misaligned semantic information and 3D localization, hence resulting in feature smearing and suboptimal predictions. To mitigate this issue, we propose ADD, an Attention-based Depth knowledge Distillation framework with 3D-aware positional encoding. Unlike previous knowledge distillation frameworks that adopt stereo- or LiDAR-based teachers, we build up our teacher with identical architecture as the student but with extra ground-truth depth as input. Credit to our teacher design, our framework is seamless, domain-gap free, easily implementable, and is compatible with object-wise ground-truth depth. Specifically, we leverage intermediate features and responses for knowledge distillation. Considering long-range 3D dependencies, we propose \emph{3D-aware self-attention} and \emph{target-aware cross-attention} modules for student adaptation. Extensive experiments are performed to verify the effectiveness of our framework on the challenging KITTI 3D object detection benchmark. We implement our framework on three representative monocular detectors, and we achieve state-of-the-art performance with no additional inference computational cost relative to baseline models. Our code is available at https://github.com/rockywind/ADD.
△ Less
Submitted 3 July, 2023; v1 submitted 30 November, 2022;
originally announced November 2022.