-
Review-Feedback-Reason (ReFeR): A Novel Framework for NLG Evaluation and Reasoning
Authors:
Yaswanth Narsupalli,
Abhranil Chandra,
Sreevatsa Muppirala,
Manish Gupta,
Pawan Goyal
Abstract:
Assessing the quality of Natural Language Generation (NLG) outputs, such as those produced by large language models (LLMs), poses significant challenges. Traditional approaches involve either resource-intensive human evaluations or automatic metrics, which often exhibit a low correlation with human judgment. In this study, we propose Review-Feedback-Reason (ReFeR), a novel evaluation framework for…
▽ More
Assessing the quality of Natural Language Generation (NLG) outputs, such as those produced by large language models (LLMs), poses significant challenges. Traditional approaches involve either resource-intensive human evaluations or automatic metrics, which often exhibit a low correlation with human judgment. In this study, we propose Review-Feedback-Reason (ReFeR), a novel evaluation framework for NLG using LLM agents. We rigorously test ReFeR using two pre-existing benchmark datasets on diverse NLG tasks. The proposed framework not only enhances the accuracy of NLG evaluation, surpassing previous benchmarks by $\sim$20\%, but also generates constructive feedback and significantly improves collective reasoning. This feedback is then leveraged for the creation of instruction-tuning datasets, which, when used to fine-tune smaller models like Mistral-7B, makes them extremely good evaluators, yielding a better correlation with human evaluations and performance nearly on par with GPT-3.5. We highlight the effectiveness of our methodology through its application on three reasoning benchmarks, where it outperforms most of the state-of-the-art methods, and also outperforms the reasoning capabilities of models like GPT-3.5 Turbo by $\sim$11.67\% and GPT-4 by $\sim$1\% on an average.
△ Less
Submitted 16 July, 2024;
originally announced July 2024.
-
Enhancing Low-Resource NMT with a Multilingual Encoder and Knowledge Distillation: A Case Study
Authors:
Aniruddha Roy,
Pretam Ray,
Ayush Maheshwari,
Sudeshna Sarkar,
Pawan Goyal
Abstract:
Neural Machine Translation (NMT) remains a formidable challenge, especially when dealing with low-resource languages. Pre-trained sequence-to-sequence (seq2seq) multi-lingual models, such as mBART-50, have demonstrated impressive performance in various low-resource NMT tasks. However, their pre-training has been confined to 50 languages, leaving out support for numerous low-resource languages, par…
▽ More
Neural Machine Translation (NMT) remains a formidable challenge, especially when dealing with low-resource languages. Pre-trained sequence-to-sequence (seq2seq) multi-lingual models, such as mBART-50, have demonstrated impressive performance in various low-resource NMT tasks. However, their pre-training has been confined to 50 languages, leaving out support for numerous low-resource languages, particularly those spoken in the Indian subcontinent. Expanding mBART-50's language support requires complex pre-training, risking performance decline due to catastrophic forgetting. Considering these expanding challenges, this paper explores a framework that leverages the benefits of a pre-trained language model along with knowledge distillation in a seq2seq architecture to facilitate translation for low-resource languages, including those not covered by mBART-50. The proposed framework employs a multilingual encoder-based seq2seq model as the foundational architecture and subsequently uses complementary knowledge distillation techniques to mitigate the impact of imbalanced training. Our framework is evaluated on three low-resource Indic languages in four Indic-to-Indic directions, yielding significant BLEU-4 and chrF improvements over baselines. Further, we conduct human evaluation to confirm effectiveness of our approach. Our code is publicly available at https://github.com/raypretam/Two-step-low-res-NMT.
△ Less
Submitted 9 July, 2024;
originally announced July 2024.
-
IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoning
Authors:
Abhinav Joshi,
Shounak Paul,
Akshat Sharma,
Pawan Goyal,
Saptarshi Ghosh,
Ashutosh Modi
Abstract:
Legal systems worldwide are inundated with exponential growth in cases and documents. There is an imminent need to develop NLP and ML techniques for automatically processing and understanding legal documents to streamline the legal system. However, evaluating and comparing various NLP models designed specifically for the legal domain is challenging. This paper addresses this challenge by proposing…
▽ More
Legal systems worldwide are inundated with exponential growth in cases and documents. There is an imminent need to develop NLP and ML techniques for automatically processing and understanding legal documents to streamline the legal system. However, evaluating and comparing various NLP models designed specifically for the legal domain is challenging. This paper addresses this challenge by proposing IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoning. IL-TUR contains monolingual (English, Hindi) and multi-lingual (9 Indian languages) domain-specific tasks that address different aspects of the legal system from the point of view of understanding and reasoning over Indian legal documents. We present baseline models (including LLM-based) for each task, outlining the gap between models and the ground truth. To foster further research in the legal domain, we create a leaderboard (available at: https://exploration-lab.github.io/IL-TUR/) where the research community can upload and compare legal text understanding systems.
△ Less
Submitted 7 July, 2024;
originally announced July 2024.
-
Challenging the Need for Packet Spraying in Large-Scale Distributed Training
Authors:
Vamsi Addanki,
Prateesh Goyal,
Ilias Marinos
Abstract:
Large-scale distributed training in production datacenters constitutes a challenging workload bottlenecked by network communication. In response, both major industry players (e.g., Ultra Ethernet Consortium) and parts of academia have surprisingly, and almost unanimously, agreed that packet spraying is necessary to improve the performance of large-scale distributed training workloads.
In this pa…
▽ More
Large-scale distributed training in production datacenters constitutes a challenging workload bottlenecked by network communication. In response, both major industry players (e.g., Ultra Ethernet Consortium) and parts of academia have surprisingly, and almost unanimously, agreed that packet spraying is necessary to improve the performance of large-scale distributed training workloads.
In this paper, we challenge this prevailing belief and pose the question: How close can a singlepath transport approach an optimal multipath transport? We demonstrate that singlepath transport (from a NIC's perspective) is sufficient and can perform nearly as well as an ideal multipath transport with packet spraying, particularly in the context of distributed training in leaf-spine topologies. Our assertion is based on four key observations about workloads driven by collective communication patterns: (i) flows within a collective start almost simultaneously, (ii) flow sizes are nearly equal, (iii) the completion time of a collective is more crucial than individual flow completion times, and (iv) flows can be split upon arrival. We analytically prove that singlepath transport, using minimal flow splitting (at the application layer), is equivalent to an ideal multipath transport with packet spraying in terms of maximum congestion. Our preliminary evaluations support our claims. This paper suggests an alternative agenda for developing next-generation transport protocols tailored for large-scale distributed training.
△ Less
Submitted 29 June, 2024;
originally announced July 2024.
-
On The Persona-based Summarization of Domain-Specific Documents
Authors:
Ankan Mullick,
Sombit Bose,
Rounak Saha,
Ayan Kumar Bhowmick,
Pawan Goyal,
Niloy Ganguly,
Prasenjit Dey,
Ravi Kokku
Abstract:
In an ever-expanding world of domain-specific knowledge, the increasing complexity of consuming, and storing information necessitates the generation of summaries from large information repositories. However, every persona of a domain has different requirements of information and hence their summarization. For example, in the healthcare domain, a persona-based (such as Doctor, Nurse, Patient etc.)…
▽ More
In an ever-expanding world of domain-specific knowledge, the increasing complexity of consuming, and storing information necessitates the generation of summaries from large information repositories. However, every persona of a domain has different requirements of information and hence their summarization. For example, in the healthcare domain, a persona-based (such as Doctor, Nurse, Patient etc.) approach is imperative to deliver targeted medical information efficiently. Persona-based summarization of domain-specific information by humans is a high cognitive load task and is generally not preferred. The summaries generated by two different humans have high variability and do not scale in cost and subject matter expertise as domains and personas grow. Further, AI-generated summaries using generic Large Language Models (LLMs) may not necessarily offer satisfactory accuracy for different domains unless they have been specifically trained on domain-specific data and can also be very expensive to use in day-to-day operations. Our contribution in this paper is two-fold: 1) We present an approach to efficiently fine-tune a domain-specific small foundation LLM using a healthcare corpus and also show that we can effectively evaluate the summarization quality using AI-based critiquing. 2) We further show that AI-based critiquing has good concordance with Human-based critiquing of the summaries. Hence, such AI-based pipelines to generate domain-specific persona-based summaries can be easily scaled to other domains such as legal, enterprise documents, education etc. in a very efficient and cost-effective manner.
△ Less
Submitted 6 June, 2024;
originally announced June 2024.
-
GS-PINN: Greedy Sampling for Parameter Estimation in Partial Differential Equations
Authors:
Ali Forootani,
Harshit Kapadia,
Sridhar Chellappa,
Pawan Goyal,
Peter Benner
Abstract:
Partial differential equation parameter estimation is a mathematical and computational process used to estimate the unknown parameters in a partial differential equation model from observational data. This paper employs a greedy sampling approach based on the Discrete Empirical Interpolation Method to identify the most informative samples in a dataset associated with a partial differential equatio…
▽ More
Partial differential equation parameter estimation is a mathematical and computational process used to estimate the unknown parameters in a partial differential equation model from observational data. This paper employs a greedy sampling approach based on the Discrete Empirical Interpolation Method to identify the most informative samples in a dataset associated with a partial differential equation to estimate its parameters. Greedy samples are used to train a physics-informed neural network architecture which maps the nonlinear relation between spatio-temporal data and the measured values. To prove the impact of greedy samples on the training of the physics-informed neural network for parameter estimation of a partial differential equation, their performance is compared with random samples taken from the given dataset. Our simulation results show that for all considered partial differential equations, greedy samples outperform random samples, i.e., we can estimate parameters with a significantly lower number of samples while simultaneously reducing the relative estimation error. A Python package is also prepared to support different phases of the proposed algorithm, including data prepossessing, greedy sampling, neural network training, and comparison.
△ Less
Submitted 14 May, 2024;
originally announced May 2024.
-
Parameter-Efficient Instruction Tuning of Large Language Models For Extreme Financial Numeral Labelling
Authors:
Subhendu Khatuya,
Rajdeep Mukherjee,
Akash Ghosh,
Manjunath Hegde,
Koustuv Dasgupta,
Niloy Ganguly,
Saptarshi Ghosh,
Pawan Goyal
Abstract:
We study the problem of automatically annotating relevant numerals (GAAP metrics) occurring in the financial documents with their corresponding XBRL tags. Different from prior works, we investigate the feasibility of solving this extreme classification problem using a generative paradigm through instruction tuning of Large Language Models (LLMs). To this end, we leverage metric metadata informatio…
▽ More
We study the problem of automatically annotating relevant numerals (GAAP metrics) occurring in the financial documents with their corresponding XBRL tags. Different from prior works, we investigate the feasibility of solving this extreme classification problem using a generative paradigm through instruction tuning of Large Language Models (LLMs). To this end, we leverage metric metadata information to frame our target outputs while proposing a parameter efficient solution for the task using LoRA. We perform experiments on two recently released financial numeric labeling datasets. Our proposed model, FLAN-FinXC, achieves new state-of-the-art performances on both the datasets, outperforming several strong baselines. We explain the better scores of our proposed model by demonstrating its capability for zero-shot as well as the least frequently occurring tags. Also, even when we fail to predict the XBRL tags correctly, our generated output has substantial overlap with the ground-truth in majority of the cases.
△ Less
Submitted 15 May, 2024; v1 submitted 3 May, 2024;
originally announced May 2024.
-
Instruction-Guided Bullet Point Summarization of Long Financial Earnings Call Transcripts
Authors:
Subhendu Khatuya,
Koushiki Sinha,
Niloy Ganguly,
Saptarshi Ghosh,
Pawan Goyal
Abstract:
While automatic summarization techniques have made significant advancements, their primary focus has been on summarizing short news articles or documents that have clear structural patterns like scientific articles or government reports. There has not been much exploration into developing efficient methods for summarizing financial documents, which often contain complex facts and figures. Here, we…
▽ More
While automatic summarization techniques have made significant advancements, their primary focus has been on summarizing short news articles or documents that have clear structural patterns like scientific articles or government reports. There has not been much exploration into developing efficient methods for summarizing financial documents, which often contain complex facts and figures. Here, we study the problem of bullet point summarization of long Earning Call Transcripts (ECTs) using the recently released ECTSum dataset. We leverage an unsupervised question-based extractive module followed by a parameter efficient instruction-tuned abstractive module to solve this task. Our proposed model FLAN-FinBPS achieves new state-of-the-art performances outperforming the strongest baseline with 14.88% average ROUGE score gain, and is capable of generating factually consistent bullet point summaries that capture the important facts discussed in the ECTs.
△ Less
Submitted 3 May, 2024;
originally announced May 2024.
-
S-EQA: Tackling Situational Queries in Embodied Question Answering
Authors:
Vishnu Sashank Dorbala,
Prasoon Goyal,
Robinson Piramuthu,
Michael Johnston,
Dinesh Manocha,
Reza Ghanadhan
Abstract:
We present and tackle the problem of Embodied Question Answering (EQA) with Situational Queries (S-EQA) in a household environment. Unlike prior EQA work tackling simple queries that directly reference target objects and quantifiable properties pertaining them, EQA with situational queries (such as "Is the bathroom clean and dry?") is more challenging, as the agent needs to figure out not just wha…
▽ More
We present and tackle the problem of Embodied Question Answering (EQA) with Situational Queries (S-EQA) in a household environment. Unlike prior EQA work tackling simple queries that directly reference target objects and quantifiable properties pertaining them, EQA with situational queries (such as "Is the bathroom clean and dry?") is more challenging, as the agent needs to figure out not just what the target objects pertaining to the query are, but also requires a consensus on their states to be answerable. Towards this objective, we first introduce a novel Prompt-Generate-Evaluate (PGE) scheme that wraps around an LLM's output to create a dataset of unique situational queries, corresponding consensus object information, and predicted answers. PGE maintains uniqueness among the generated queries, using multiple forms of semantic similarity. We validate the generated dataset via a large scale user-study conducted on M-Turk, and introduce it as S-EQA, the first dataset tackling EQA with situational queries. Our user study establishes the authenticity of S-EQA with a high 97.26% of the generated queries being deemed answerable, given the consensus object data. Conversely, we observe a low correlation of 46.2% on the LLM-predicted answers to human-evaluated ones; indicating the LLM's poor capability in directly answering situational queries, while establishing S-EQA's usability in providing a human-validated consensus for an indirect solution. We evaluate S-EQA via Visual Question Answering (VQA) on VirtualHome, which unlike other simulators, contains several objects with modifiable states that also visually appear different upon modification -- enabling us to set a quantitative benchmark for S-EQA. To the best of our knowledge, this is the first work to introduce EQA with situational queries, and also the first to use a generative approach for query creation.
△ Less
Submitted 7 May, 2024;
originally announced May 2024.
-
LTLDoG: Satisfying Temporally-Extended Symbolic Constraints for Safe Diffusion-based Planning
Authors:
Zeyu Feng,
Hao Luan,
Pranav Goyal,
Harold Soh
Abstract:
Operating effectively in complex environments while complying with specified constraints is crucial for the safe and successful deployment of robots that interact with and operate around people. In this work, we focus on generating long-horizon trajectories that adhere to novel static and temporally-extended constraints/instructions at test time. We propose a data-driven diffusion-based framework,…
▽ More
Operating effectively in complex environments while complying with specified constraints is crucial for the safe and successful deployment of robots that interact with and operate around people. In this work, we focus on generating long-horizon trajectories that adhere to novel static and temporally-extended constraints/instructions at test time. We propose a data-driven diffusion-based framework, LTLDoG, that modifies the inference steps of the reverse process given an instruction specified using finite linear temporal logic ($\text{LTL}_f$). LTLDoG leverages a satisfaction value function on $\text{LTL}_f$ and guides the sampling steps using its gradient field. This value function can also be trained to generalize to new instructions not observed during training, enabling flexible test-time adaptability. Experiments in robot navigation and manipulation illustrate that the method is able to generate trajectories that satisfy formulae that specify obstacle avoidance and visitation sequences.
△ Less
Submitted 7 May, 2024;
originally announced May 2024.
-
SERPENT-VLM : Self-Refining Radiology Report Generation Using Vision Language Models
Authors:
Manav Nitin Kapadnis,
Sohan Patnaik,
Abhilash Nandy,
Sourjyadip Ray,
Pawan Goyal,
Debdoot Sheet
Abstract:
Radiology Report Generation (R2Gen) demonstrates how Multi-modal Large Language Models (MLLMs) can automate the creation of accurate and coherent radiological reports. Existing methods often hallucinate details in text-based reports that don't accurately reflect the image content. To mitigate this, we introduce a novel strategy, SERPENT-VLM (SElf Refining Radiology RePort GENeraTion using Vision L…
▽ More
Radiology Report Generation (R2Gen) demonstrates how Multi-modal Large Language Models (MLLMs) can automate the creation of accurate and coherent radiological reports. Existing methods often hallucinate details in text-based reports that don't accurately reflect the image content. To mitigate this, we introduce a novel strategy, SERPENT-VLM (SElf Refining Radiology RePort GENeraTion using Vision Language Models), which improves the R2Gen task by integrating a self-refining mechanism into the MLLM framework. We employ a unique self-supervised loss that leverages similarity between pooled image representations and the contextual representations of the generated radiological text, alongside the standard Causal Language Modeling objective, to refine image-text representations. This allows the model to scrutinize and align the generated text through dynamic interaction between a given image and the generated text, therefore reducing hallucination and continuously enhancing nuanced report generation. SERPENT-VLM outperforms existing baselines such as LLaVA-Med, BiomedGPT, etc., achieving SoTA performance on the IU X-ray and Radiology Objects in COntext (ROCO) datasets, and also proves to be robust against noisy images. A qualitative case study emphasizes the significant advancements towards more sophisticated MLLM frameworks for R2Gen, opening paths for further research into self-supervised refinement in the medical imaging domain.
△ Less
Submitted 18 July, 2024; v1 submitted 27 April, 2024;
originally announced April 2024.
-
Investigating Cosmic Homogeneity Using Multi-fractal Analysis of the SDSS-IV eBOSS DR16 Quasar Catalog
Authors:
Priya Goyal,
Sunil Malik,
Jaswant k. Yadav,
T. R. Seshadri
Abstract:
We analyze the volume-limited subsamples extracted from the sixteenth data release of the SDSS-IV eBOSS quasar survey spanning a redshift interval of $0.8 < z < 2.2$, to estimate the scale of transition to homogeneity in the Universe. The multi-fractal analysis used for this purpose considers the scaling behavior of different moments of quasar distribution in different density environments. This a…
▽ More
We analyze the volume-limited subsamples extracted from the sixteenth data release of the SDSS-IV eBOSS quasar survey spanning a redshift interval of $0.8 < z < 2.2$, to estimate the scale of transition to homogeneity in the Universe. The multi-fractal analysis used for this purpose considers the scaling behavior of different moments of quasar distribution in different density environments. This analysis gives the spectrum of generalized dimension $D_q$, where positive values of $q$ characterize the scaling behavior in over-dense regions and the negative ones in under-dense regions. We expect fractal correlation dimension $D_q(r) = 3$, for a homogeneous, random point distribution in 3-Dimensions. The fractal correlation dimension $D_q(r)$, corresponding to $q=2$ obtained in our study stabilizes in the range (2.8-2.9) for scales $r>80$ $h^{-1}$ Mpc. The observed quasar distribution shows consistency with the simulated mock data and the random distribution of quasars within one sigma. Further, the generalized dimension spectrum $D_q(r)$ also reveals transition to homogeneity beyond $>110$ $h^{-1}$ Mpc, and the dominance of clustering at small scales $r<80$ $h^{-1}$ Mpc. Consequently, our study provides strong evidence for the homogeneity in SDSS quasar distribution, offering insights into large-scale structure properties and, thus can play a pivotal role in scrutinizing the clustering properties of quasars and its evolution in various upcoming surveys such as Dark Energy Spectroscopic Instrument (DESI) and Extremely Large Telescope (ELT).
△ Less
Submitted 14 April, 2024;
originally announced April 2024.
-
"Don't forget to put the milk back!" Dataset for Enabling Embodied Agents to Detect Anomalous Situations
Authors:
James F. Mullen Jr,
Prasoon Goyal,
Robinson Piramuthu,
Michael Johnston,
Dinesh Manocha,
Reza Ghanadan
Abstract:
Home robots intend to make their users lives easier. Our work assists in this goal by enabling robots to inform their users of dangerous or unsanitary anomalies in their home. Some examples of these anomalies include the user leaving their milk out, forgetting to turn off the stove, or leaving poison accessible to children. To move towards enabling home robots with these abilities, we have created…
▽ More
Home robots intend to make their users lives easier. Our work assists in this goal by enabling robots to inform their users of dangerous or unsanitary anomalies in their home. Some examples of these anomalies include the user leaving their milk out, forgetting to turn off the stove, or leaving poison accessible to children. To move towards enabling home robots with these abilities, we have created a new dataset, which we call SafetyDetect. The SafetyDetect dataset consists of 1000 anomalous home scenes, each of which contains unsafe or unsanitary situations for an agent to detect. Our approach utilizes large language models (LLMs) alongside both a graph representation of the scene and the relationships between the objects in the scene. Our key insight is that this connected scene graph and the object relationships it encodes enables the LLM to better reason about the scene -- especially as it relates to detecting dangerous or unsanitary situations. Our most promising approach utilizes GPT-4 and pursues a categorization technique where object relations from the scene graph are classified as normal, dangerous, unsanitary, or dangerous for children. This method is able to correctly identify over 90% of anomalous scenarios in the SafetyDetect Dataset. Additionally, we conduct real world experiments on a ClearPath TurtleBot where we generate a scene graph from visuals of the real world scene, and run our approach with no modification. This setup resulted in little performance loss. The SafetyDetect Dataset and code will be released to the public upon this papers publication.
△ Less
Submitted 12 April, 2024;
originally announced April 2024.
-
Order-Based Pre-training Strategies for Procedural Text Understanding
Authors:
Abhilash Nandy,
Yash Kulkarni,
Pawan Goyal,
Niloy Ganguly
Abstract:
In this paper, we propose sequence-based pretraining methods to enhance procedural understanding in natural language processing. Procedural text, containing sequential instructions to accomplish a task, is difficult to understand due to the changing attributes of entities in the context. We focus on recipes, which are commonly represented as ordered instructions, and use this order as a supervisio…
▽ More
In this paper, we propose sequence-based pretraining methods to enhance procedural understanding in natural language processing. Procedural text, containing sequential instructions to accomplish a task, is difficult to understand due to the changing attributes of entities in the context. We focus on recipes, which are commonly represented as ordered instructions, and use this order as a supervision signal. Our work is one of the first to compare several 'order as-supervision' transformer pre-training methods, including Permutation Classification, Embedding Regression, and Skip-Clip, and shows that these methods give improved results compared to the baselines and SoTA LLMs on two downstream Entity-Tracking datasets: NPN-Cooking dataset in recipe domain and ProPara dataset in open domain. Our proposed methods address the non-trivial Entity Tracking Task that requires prediction of entity states across procedure steps, which requires understanding the order of steps. These methods show an improvement over the best baseline by 1.6% and 7-9% on NPN-Cooking and ProPara Datasets respectively across metrics.
△ Less
Submitted 6 April, 2024;
originally announced April 2024.
-
Intent Detection and Entity Extraction from BioMedical Literature
Authors:
Ankan Mullick,
Mukur Gupta,
Pawan Goyal
Abstract:
Biomedical queries have become increasingly prevalent in web searches, reflecting the growing interest in accessing biomedical literature. Despite recent research on large-language models (LLMs) motivated by endeavours to attain generalized intelligence, their efficacy in replacing task and domain-specific natural language understanding approaches remains questionable. In this paper, we address th…
▽ More
Biomedical queries have become increasingly prevalent in web searches, reflecting the growing interest in accessing biomedical literature. Despite recent research on large-language models (LLMs) motivated by endeavours to attain generalized intelligence, their efficacy in replacing task and domain-specific natural language understanding approaches remains questionable. In this paper, we address this question by conducting a comprehensive empirical evaluation of intent detection and named entity recognition (NER) tasks from biomedical text. We show that Supervised Fine Tuned approaches are still relevant and more effective than general-purpose LLMs. Biomedical transformer models such as PubMedBERT can surpass ChatGPT on NER task with only 5 supervised examples.
△ Less
Submitted 4 April, 2024;
originally announced April 2024.
-
How Robust are the Tabular QA Models for Scientific Tables? A Study using Customized Dataset
Authors:
Akash Ghosh,
B Venkata Sahith,
Niloy Ganguly,
Pawan Goyal,
Mayank Singh
Abstract:
Question-answering (QA) on hybrid scientific tabular and textual data deals with scientific information, and relies on complex numerical reasoning. In recent years, while tabular QA has seen rapid progress, understanding their robustness on scientific information is lacking due to absence of any benchmark dataset. To investigate the robustness of the existing state-of-the-art QA models on scientif…
▽ More
Question-answering (QA) on hybrid scientific tabular and textual data deals with scientific information, and relies on complex numerical reasoning. In recent years, while tabular QA has seen rapid progress, understanding their robustness on scientific information is lacking due to absence of any benchmark dataset. To investigate the robustness of the existing state-of-the-art QA models on scientific hybrid tabular data, we propose a new dataset, "SciTabQA", consisting of 822 question-answer pairs from scientific tables and their descriptions. With the help of this dataset, we assess the state-of-the-art Tabular QA models based on their ability (i) to use heterogeneous information requiring both structured data (table) and unstructured data (text) and (ii) to perform complex scientific reasoning tasks. In essence, we check the capability of the models to interpret scientific tables and text. Our experiments show that "SciTabQA" is an innovative dataset to study question-answering over scientific heterogeneous data. We benchmark three state-of-the-art Tabular QA models, and find that the best F1 score is only 0.462.
△ Less
Submitted 30 March, 2024;
originally announced April 2024.
-
CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?
Authors:
Ibrahim Alabdulmohsin,
Xiao Wang,
Andreas Steiner,
Priya Goyal,
Alexander D'Amour,
Xiaohua Zhai
Abstract:
We study the effectiveness of data-balancing for mitigating biases in contrastive language-image pretraining (CLIP), identifying areas of strength and limitation. First, we reaffirm prior conclusions that CLIP models can inadvertently absorb societal stereotypes. To counter this, we present a novel algorithm, called Multi-Modal Moment Matching (M4), designed to reduce both representation and assoc…
▽ More
We study the effectiveness of data-balancing for mitigating biases in contrastive language-image pretraining (CLIP), identifying areas of strength and limitation. First, we reaffirm prior conclusions that CLIP models can inadvertently absorb societal stereotypes. To counter this, we present a novel algorithm, called Multi-Modal Moment Matching (M4), designed to reduce both representation and association biases (i.e. in first- and second-order statistics) in multimodal data. We use M4 to conduct an in-depth analysis taking into account various factors, such as the model, representation, and data size. Our study also explores the dynamic nature of how CLIP learns and unlearns biases. In particular, we find that fine-tuning is effective in countering representation biases, though its impact diminishes for association biases. Also, data balancing has a mixed impact on quality: it tends to improve classification but can hurt retrieval. Interestingly, data and architectural improvements seem to mitigate the negative impact of data balancing on performance; e.g. applying M4 to SigLIP-B/16 with data quality filters improves COCO image-to-text retrieval @5 from 86% (without data balancing) to 87% and ImageNet 0-shot classification from 77% to 77.5%! Finally, we conclude with recommendations for improving the efficacy of data balancing in multimodal systems.
△ Less
Submitted 7 March, 2024;
originally announced March 2024.
-
Stability-Certified Learning of Control Systems with Quadratic Nonlinearities
Authors:
Igor Pontes Duff,
Pawan Goyal,
Peter Benner
Abstract:
This work primarily focuses on an operator inference methodology aimed at constructing low-dimensional dynamical models based on a priori hypotheses about their structure, often informed by established physics or expert insights. Stability is a fundamental attribute of dynamical systems, yet it is not always assured in models derived through inference. Our main objective is to develop a method tha…
▽ More
This work primarily focuses on an operator inference methodology aimed at constructing low-dimensional dynamical models based on a priori hypotheses about their structure, often informed by established physics or expert insights. Stability is a fundamental attribute of dynamical systems, yet it is not always assured in models derived through inference. Our main objective is to develop a method that facilitates the inference of quadratic control dynamical systems with inherent stability guarantees. To this aim, we investigate the stability characteristics of control systems with energy-preserving nonlinearities, thereby identifying conditions under which such systems are bounded-input bounded-state stable. These insights are subsequently applied to the learning process, yielding inferred models that are inherently stable by design. The efficacy of our proposed framework is demonstrated through a couple of numerical examples.
△ Less
Submitted 1 March, 2024;
originally announced March 2024.
-
The Structure of Quantum Questions
Authors:
Yang Yu,
Philip Goyal
Abstract:
In classical physics, a single measurement can in principle reveal the state of a system. However, quantum theory permits numerous non-equivalent measurements on a physical system, each providing only limited information about the state. This set of various measurements on a quantum system indicates a rich internal structure. We illuminate this structure for both individual and composite systems b…
▽ More
In classical physics, a single measurement can in principle reveal the state of a system. However, quantum theory permits numerous non-equivalent measurements on a physical system, each providing only limited information about the state. This set of various measurements on a quantum system indicates a rich internal structure. We illuminate this structure for both individual and composite systems by conceptualizing measurements as questions with a finite number of outcomes. We create a mathematical question structure to explore the underlying properties, employing the concept of information as a key tool representing our knowledge gained from asking these questions. We subsequently propose informational assumptions based on properties observed from measurements on qubits, generalizing these to higher dimensional systems.
Our informational assumptions shape the correlations between subsystems, which are symbolized as classical logical gates. Interestingly, systems with prime number dimensions exhibit unique property: the logical gate can be expressed simply as a linear equation under modular arithmetic. We also identify structures in quantum theory that correspond to those in the structure of quantum questions. For instance, the questions determining the system correspond to generalized Pauli matrices, and the logical gate connecting questions in subsystems is directly related to the tensor product combining operators. Based on these correspondences, we present two equivalent scenarios regarding the evolution of systems and the change of information within both quantum questions and quantum mechanics.
△ Less
Submitted 29 February, 2024;
originally announced February 2024.
-
Learning reduced-order Quadratic-Linear models in Process Engineering using Operator Inference
Authors:
Ion Victor Gosea,
Luisa Peterson,
Pawan Goyal,
Jens Bremer,
Kai Sundmacher,
Peter Benner
Abstract:
In this work, we address the challenge of efficiently modeling dynamical systems in process engineering. We use reduced-order model learning, specifically operator inference. This is a non-intrusive, data-driven method for learning dynamical systems from time-domain data. The application in our study is carbon dioxide methanation, an important reaction within the Power-to-X framework, to demonstra…
▽ More
In this work, we address the challenge of efficiently modeling dynamical systems in process engineering. We use reduced-order model learning, specifically operator inference. This is a non-intrusive, data-driven method for learning dynamical systems from time-domain data. The application in our study is carbon dioxide methanation, an important reaction within the Power-to-X framework, to demonstrate its potential. The numerical results show the ability of the reduced-order models constructed with operator inference to provide a reduced yet accurate surrogate solution. This represents an important milestone towards the implementation of fast and reliable digital twin architectures.
△ Less
Submitted 30 July, 2024; v1 submitted 27 February, 2024;
originally announced February 2024.
-
GEM3D: GEnerative Medial Abstractions for 3D Shape Synthesis
Authors:
Dmitry Petrov,
Pradyumn Goyal,
Vikas Thamizharasan,
Vladimir G. Kim,
Matheus Gadelha,
Melinos Averkiou,
Siddhartha Chaudhuri,
Evangelos Kalogerakis
Abstract:
We introduce GEM3D -- a new deep, topology-aware generative model of 3D shapes. The key ingredient of our method is a neural skeleton-based representation encoding information on both shape topology and geometry. Through a denoising diffusion probabilistic model, our method first generates skeleton-based representations following the Medial Axis Transform (MAT), then generates surfaces through a s…
▽ More
We introduce GEM3D -- a new deep, topology-aware generative model of 3D shapes. The key ingredient of our method is a neural skeleton-based representation encoding information on both shape topology and geometry. Through a denoising diffusion probabilistic model, our method first generates skeleton-based representations following the Medial Axis Transform (MAT), then generates surfaces through a skeleton-driven neural implicit formulation. The neural implicit takes into account the topological and geometric information stored in the generated skeleton representations to yield surfaces that are more topologically and geometrically accurate compared to previous neural field formulations. We discuss applications of our method in shape synthesis and point cloud reconstruction tasks, and evaluate our method both qualitatively and quantitatively. We demonstrate significantly more faithful surface reconstruction and diverse shape generation results compared to the state-of-the-art, also involving challenging scenarios of reconstructing and synthesizing structurally complex, high-genus shape surfaces from Thingi10K and ShapeNet.
△ Less
Submitted 10 April, 2024; v1 submitted 26 February, 2024;
originally announced February 2024.
-
Long Dialog Summarization: An Analysis
Authors:
Ankan Mullick,
Ayan Kumar Bhowmick,
Raghav R,
Ravi Kokku,
Prasenjit Dey,
Pawan Goyal,
Niloy Ganguly
Abstract:
Dialog summarization has become increasingly important in managing and comprehending large-scale conversations across various domains. This task presents unique challenges in capturing the key points, context, and nuances of multi-turn long conversations for summarization. It is worth noting that the summarization techniques may vary based on specific requirements such as in a shopping-chatbot sce…
▽ More
Dialog summarization has become increasingly important in managing and comprehending large-scale conversations across various domains. This task presents unique challenges in capturing the key points, context, and nuances of multi-turn long conversations for summarization. It is worth noting that the summarization techniques may vary based on specific requirements such as in a shopping-chatbot scenario, the dialog summary helps to learn user preferences, whereas in the case of a customer call center, the summary may involve the problem attributes that a user specified, and the final resolution provided. This work emphasizes the significance of creating coherent and contextually rich summaries for effective communication in various applications. We explore current state-of-the-art approaches for long dialog summarization in different domains and benchmark metrics based evaluations show that one single model does not perform well across various areas for distinct summarization tasks.
△ Less
Submitted 26 February, 2024;
originally announced February 2024.
-
Balanced Truncation of Descriptor Systems with a Quadratic Output
Authors:
Jennifer Przybilla,
Igor Pontes Duff,
Pawan Goyal,
Peter Benner
Abstract:
This work discusses model reduction for differential-algebraic systems with quadratic output equations. Under mild conditions, these systems can be transformed into a Weierstraß canonical form and, thus, be decoupled into differential equations and algebraic equations. The corresponding decoupled states are referred to as proper and improper states. Due to the quadratic function of the state as an…
▽ More
This work discusses model reduction for differential-algebraic systems with quadratic output equations. Under mild conditions, these systems can be transformed into a Weierstraß canonical form and, thus, be decoupled into differential equations and algebraic equations. The corresponding decoupled states are referred to as proper and improper states. Due to the quadratic function of the state as an output, the proper and improper states are coupled in the output equation, which imposes a challenge from a model reduction viewpoint. Keeping the coupling in mind, our goal in this work is to find important subspaces of the proper and improper states and to reduce the system accordingly. To that end, we first propose the system's matrices, the so-called Gramians, to characterize the system's dominant subspaces. We pay particular attention to the computation of the observability Gramians that take into account the nonlinear coupling between the proper and the improper states. We furthermore show that the proposed Gramians are related to certain kernel functions, which are used to identify important subspaces. This allows us to propose a reduction algorithm to obtain reduced-order systems by removing the subspaces that are difficult to reach, as well as, difficult to observe. Moreover, we quantify the error between the full-order and reduced-order models and demonstrate the proposed methodology using three numerical experiments.
△ Less
Submitted 22 February, 2024;
originally announced February 2024.
-
Roadmap on Data-Centric Materials Science
Authors:
Stefan Bauer,
Peter Benner,
Tristan Bereau,
Volker Blum,
Mario Boley,
Christian Carbogno,
C. Richard A. Catlow,
Gerhard Dehm,
Sebastian Eibl,
Ralph Ernstorfer,
Ádám Fekete,
Lucas Foppa,
Peter Fratzl,
Christoph Freysoldt,
Baptiste Gault,
Luca M. Ghiringhelli,
Sajal K. Giri,
Anton Gladyshev,
Pawan Goyal,
Jason Hattrick-Simpers,
Lara Kabalan,
Petr Karpov,
Mohammad S. Khorrami,
Christoph Koch,
Sebastian Kokott
, et al. (36 additional authors not shown)
Abstract:
Science is and always has been based on data, but the terms "data-centric" and the "4th paradigm of" materials research indicate a radical change in how information is retrieved, handled and research is performed. It signifies a transformative shift towards managing vast data collections, digital repositories, and innovative data analytics methods. The integration of Artificial Intelligence (AI) a…
▽ More
Science is and always has been based on data, but the terms "data-centric" and the "4th paradigm of" materials research indicate a radical change in how information is retrieved, handled and research is performed. It signifies a transformative shift towards managing vast data collections, digital repositories, and innovative data analytics methods. The integration of Artificial Intelligence (AI) and its subset Machine Learning (ML), has become pivotal in addressing all these challenges. This Roadmap on Data-Centric Materials Science explores fundamental concepts and methodologies, illustrating diverse applications in electronic-structure theory, soft matter theory, microstructure research, and experimental techniques like photoemission, atom probe tomography, and electron microscopy. While the roadmap delves into specific areas within the broad interdisciplinary field of materials science, the provided examples elucidate key concepts applicable to a wider range of topics. The discussed instances offer insights into addressing the multifaceted challenges encountered in contemporary materials research.
△ Less
Submitted 1 May, 2024; v1 submitted 1 February, 2024;
originally announced February 2024.
-
MatSciRE: Leveraging Pointer Networks to Automate Entity and Relation Extraction for Material Science Knowledge-base Construction
Authors:
Ankan Mullick,
Akash Ghosh,
G Sai Chaitanya,
Samir Ghui,
Tapas Nayak,
Seung-Cheol Lee,
Satadeep Bhattacharjee,
Pawan Goyal
Abstract:
Material science literature is a rich source of factual information about various categories of entities (like materials and compositions) and various relations between these entities, such as conductivity, voltage, etc. Automatically extracting this information to generate a material science knowledge base is a challenging task. In this paper, we propose MatSciRE (Material Science Relation Extrac…
▽ More
Material science literature is a rich source of factual information about various categories of entities (like materials and compositions) and various relations between these entities, such as conductivity, voltage, etc. Automatically extracting this information to generate a material science knowledge base is a challenging task. In this paper, we propose MatSciRE (Material Science Relation Extractor), a Pointer Network-based encoder-decoder framework, to jointly extract entities and relations from material science articles as a triplet ($entity1, relation, entity2$). Specifically, we target the battery materials and identify five relations to work on - conductivity, coulombic efficiency, capacity, voltage, and energy. Our proposed approach achieved a much better F1-score (0.771) than a previous attempt using ChemDataExtractor (0.716). The overall graphical framework of MatSciRE is shown in Fig 1. The material information is extracted from material science literature in the form of entity-relation triplets using MatSciRE.
△ Less
Submitted 18 January, 2024;
originally announced January 2024.
-
Faithful Model Evaluation for Model-Based Metrics
Authors:
Palash Goyal,
Qian Hu,
Rahul Gupta
Abstract:
Statistical significance testing is used in natural language processing (NLP) to determine whether the results of a study or experiment are likely to be due to chance or if they reflect a genuine relationship. A key step in significance testing is the estimation of confidence interval which is a function of sample variance. Sample variance calculation is straightforward when evaluating against gro…
▽ More
Statistical significance testing is used in natural language processing (NLP) to determine whether the results of a study or experiment are likely to be due to chance or if they reflect a genuine relationship. A key step in significance testing is the estimation of confidence interval which is a function of sample variance. Sample variance calculation is straightforward when evaluating against ground truth. However, in many cases, a metric model is often used for evaluation. For example, to compare toxicity of two large language models, a toxicity classifier is used for evaluation. Existing works usually do not consider the variance change due to metric model errors, which can lead to wrong conclusions. In this work, we establish the mathematical foundation of significance testing for model-based metrics. With experiments on public benchmark datasets and a production system, we show that considering metric model errors to calculate sample variances for model-based metrics changes the conclusions in certain experiments.
△ Less
Submitted 19 December, 2023;
originally announced December 2023.
-
Tokenization Matters: Navigating Data-Scarce Tokenization for Gender Inclusive Language Technologies
Authors:
Anaelia Ovalle,
Ninareh Mehrabi,
Palash Goyal,
Jwala Dhamala,
Kai-Wei Chang,
Richard Zemel,
Aram Galstyan,
Yuval Pinter,
Rahul Gupta
Abstract:
Gender-inclusive NLP research has documented the harmful limitations of gender binary-centric large language models (LLM), such as the inability to correctly use gender-diverse English neopronouns (e.g., xe, zir, fae). While data scarcity is a known culprit, the precise mechanisms through which scarcity affects this behavior remain underexplored. We discover LLM misgendering is significantly influ…
▽ More
Gender-inclusive NLP research has documented the harmful limitations of gender binary-centric large language models (LLM), such as the inability to correctly use gender-diverse English neopronouns (e.g., xe, zir, fae). While data scarcity is a known culprit, the precise mechanisms through which scarcity affects this behavior remain underexplored. We discover LLM misgendering is significantly influenced by Byte-Pair Encoding (BPE) tokenization, the tokenizer powering many popular LLMs. Unlike binary pronouns, BPE overfragments neopronouns, a direct consequence of data scarcity during tokenizer training. This disparate tokenization mirrors tokenizer limitations observed in multilingual and low-resource NLP, unlocking new misgendering mitigation strategies. We propose two techniques: (1) pronoun tokenization parity, a method to enforce consistent tokenization across gendered pronouns, and (2) utilizing pre-existing LLM pronoun knowledge to improve neopronoun proficiency. Our proposed methods outperform finetuning with standard BPE, improving neopronoun accuracy from 14.1% to 58.4%. Our paper is the first to link LLM misgendering to tokenization and deficient neopronoun grammar, indicating that LLMs unable to correctly treat neopronouns as pronouns are more prone to misgender.
△ Less
Submitted 6 April, 2024; v1 submitted 18 December, 2023;
originally announced December 2023.
-
Hate Speech and Offensive Content Detection in Indo-Aryan Languages: A Battle of LSTM and Transformers
Authors:
Nikhil Narayan,
Mrutyunjay Biswal,
Pramod Goyal,
Abhranta Panigrahi
Abstract:
Social media platforms serve as accessible outlets for individuals to express their thoughts and experiences, resulting in an influx of user-generated data spanning all age groups. While these platforms enable free expression, they also present significant challenges, including the proliferation of hate speech and offensive content. Such objectionable language disrupts objective discourse and can…
▽ More
Social media platforms serve as accessible outlets for individuals to express their thoughts and experiences, resulting in an influx of user-generated data spanning all age groups. While these platforms enable free expression, they also present significant challenges, including the proliferation of hate speech and offensive content. Such objectionable language disrupts objective discourse and can lead to radicalization of debates, ultimately threatening democratic values. Consequently, organizations have taken steps to monitor and curb abusive behavior, necessitating automated methods for identifying suspicious posts. This paper contributes to Hate Speech and Offensive Content Identification in English and Indo-Aryan Languages (HASOC) 2023 shared tasks track. We, team Z-AGI Labs, conduct a comprehensive comparative analysis of hate speech classification across five distinct languages: Bengali, Assamese, Bodo, Sinhala, and Gujarati. Our study encompasses a wide range of pre-trained models, including Bert variants, XLM-R, and LSTM models, to assess their performance in identifying hate speech across these languages. Results reveal intriguing variations in model performance. Notably, Bert Base Multilingual Cased emerges as a strong performer across languages, achieving an F1 score of 0.67027 for Bengali and 0.70525 for Assamese. At the same time, it significantly outperforms other models with an impressive F1 score of 0.83009 for Bodo. In Sinhala, XLM-R stands out with an F1 score of 0.83493, whereas for Gujarati, a custom LSTM-based model outshined with an F1 score of 0.76601. This study offers valuable insights into the suitability of various pre-trained models for hate speech detection in multilingual settings. By considering the nuances of each, our research contributes to an informed model selection for building robust hate speech detection systems.
△ Less
Submitted 9 December, 2023;
originally announced December 2023.
-
JAB: Joint Adversarial Prompting and Belief Augmentation
Authors:
Ninareh Mehrabi,
Palash Goyal,
Anil Ramakrishna,
Jwala Dhamala,
Shalini Ghosh,
Richard Zemel,
Kai-Wei Chang,
Aram Galstyan,
Rahul Gupta
Abstract:
With the recent surge of language models in different applications, attention to safety and robustness of these models has gained significant importance. Here we introduce a joint framework in which we simultaneously probe and improve the robustness of a black-box target model via adversarial prompting and belief augmentation using iterative feedback loops. This framework utilizes an automated red…
▽ More
With the recent surge of language models in different applications, attention to safety and robustness of these models has gained significant importance. Here we introduce a joint framework in which we simultaneously probe and improve the robustness of a black-box target model via adversarial prompting and belief augmentation using iterative feedback loops. This framework utilizes an automated red teaming approach to probe the target model, along with a belief augmenter to generate instructions for the target model to improve its robustness to those adversarial probes. Importantly, the adversarial model and the belief generator leverage the feedback from past interactions to improve the effectiveness of the adversarial prompts and beliefs, respectively. In our experiments, we demonstrate that such a framework can reduce toxic content generation both in dynamic cases where an adversary directly interacts with a target model and static cases where we use a static benchmark dataset to evaluate our model.
△ Less
Submitted 15 November, 2023;
originally announced November 2023.
-
On the steerability of large language models toward data-driven personas
Authors:
Junyi Li,
Ninareh Mehrabi,
Charith Peris,
Palash Goyal,
Kai-Wei Chang,
Aram Galstyan,
Richard Zemel,
Rahul Gupta
Abstract:
Large language models (LLMs) are known to generate biased responses where the opinions of certain groups and populations are underrepresented. Here, we present a novel approach to achieve controllable generation of specific viewpoints using LLMs, that can be leveraged to produce multiple perspectives and to reflect the diverse opinions. Moving beyond the traditional reliance on demographics like a…
▽ More
Large language models (LLMs) are known to generate biased responses where the opinions of certain groups and populations are underrepresented. Here, we present a novel approach to achieve controllable generation of specific viewpoints using LLMs, that can be leveraged to produce multiple perspectives and to reflect the diverse opinions. Moving beyond the traditional reliance on demographics like age, gender, or party affiliation, we introduce a data-driven notion of persona grounded in collaborative filtering, which is defined as either a single individual or a cohort of individuals manifesting similar views across specific inquiries. As individuals in the same demographic group may have different personas, our data-driven persona definition allows for a more nuanced understanding of different (latent) social groups present in the population. In addition to this, we also explore an efficient method to steer LLMs toward the personas that we define. We show that our data-driven personas significantly enhance model steerability, with improvements of between $57\%-77\%$ over our best performing baselines.
△ Less
Submitted 2 April, 2024; v1 submitted 8 November, 2023;
originally announced November 2023.
-
Extracting Entities of Interest from Comparative Product Reviews
Authors:
Jatin Arora,
Sumit Agrawal,
Pawan Goyal,
Sayan Pathak
Abstract:
This paper presents a deep learning based approach to extract product comparison information out of user reviews on various e-commerce websites. Any comparative product review has three major entities of information: the names of the products being compared, the user opinion (predicate) and the feature or aspect under comparison. All these informing entities are dependent on each other and bound b…
▽ More
This paper presents a deep learning based approach to extract product comparison information out of user reviews on various e-commerce websites. Any comparative product review has three major entities of information: the names of the products being compared, the user opinion (predicate) and the feature or aspect under comparison. All these informing entities are dependent on each other and bound by the rules of the language, in the review. We observe that their inter-dependencies can be captured well using LSTMs. We evaluate our system on existing manually labeled datasets and observe out-performance over the existing Semantic Role Labeling (SRL) framework popular for this task.
△ Less
Submitted 31 October, 2023;
originally announced October 2023.
-
CONTRASTE: Supervised Contrastive Pre-training With Aspect-based Prompts For Aspect Sentiment Triplet Extraction
Authors:
Rajdeep Mukherjee,
Nithish Kannen,
Saurabh Kumar Pandey,
Pawan Goyal
Abstract:
Existing works on Aspect Sentiment Triplet Extraction (ASTE) explicitly focus on developing more efficient fine-tuning techniques for the task. Instead, our motivation is to come up with a generic approach that can improve the downstream performances of multiple ABSA tasks simultaneously. Towards this, we present CONTRASTE, a novel pre-training strategy using CONTRastive learning to enhance the AS…
▽ More
Existing works on Aspect Sentiment Triplet Extraction (ASTE) explicitly focus on developing more efficient fine-tuning techniques for the task. Instead, our motivation is to come up with a generic approach that can improve the downstream performances of multiple ABSA tasks simultaneously. Towards this, we present CONTRASTE, a novel pre-training strategy using CONTRastive learning to enhance the ASTE performance. While we primarily focus on ASTE, we also demonstrate the advantage of our proposed technique on other ABSA tasks such as ACOS, TASD, and AESC. Given a sentence and its associated (aspect, opinion, sentiment) triplets, first, we design aspect-based prompts with corresponding sentiments masked. We then (pre)train an encoder-decoder model by applying contrastive learning on the decoder-generated aspect-aware sentiment representations of the masked terms. For fine-tuning the model weights thus obtained, we then propose a novel multi-task approach where the base encoder-decoder model is combined with two complementary modules, a tagging-based Opinion Term Detector, and a regression-based Triplet Count Estimator. Exhaustive experiments on four benchmark datasets and a detailed ablation study establish the importance of each of our proposed components as we achieve new state-of-the-art ASTE results.
△ Less
Submitted 24 October, 2023;
originally announced October 2023.
-
CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Text
Authors:
Abhilash Nandy,
Manav Nitin Kapadnis,
Pawan Goyal,
Niloy Ganguly
Abstract:
In this paper, we propose CLMSM, a domain-specific, continual pre-training framework, that learns from a large set of procedural recipes. CLMSM uses a Multi-Task Learning Framework to optimize two objectives - a) Contrastive Learning using hard triplets to learn fine-grained differences across entities in the procedures, and b) a novel Mask-Step Modelling objective to learn step-wise context of a…
▽ More
In this paper, we propose CLMSM, a domain-specific, continual pre-training framework, that learns from a large set of procedural recipes. CLMSM uses a Multi-Task Learning Framework to optimize two objectives - a) Contrastive Learning using hard triplets to learn fine-grained differences across entities in the procedures, and b) a novel Mask-Step Modelling objective to learn step-wise context of a procedure. We test the performance of CLMSM on the downstream tasks of tracking entities and aligning actions between two procedures on three datasets, one of which is an open-domain dataset not conforming with the pre-training dataset. We show that CLMSM not only outperforms baselines on recipes (in-domain) but is also able to generalize to open-domain procedural NLP tasks.
△ Less
Submitted 22 October, 2023;
originally announced October 2023.
-
DepNeCTI: Dependency-based Nested Compound Type Identification for Sanskrit
Authors:
Jivnesh Sandhan,
Yaswanth Narsupalli,
Sreevatsa Muppirala,
Sriram Krishnan,
Pavankumar Satuluri,
Amba Kulkarni,
Pawan Goyal
Abstract:
Multi-component compounding is a prevalent phenomenon in Sanskrit, and understanding the implicit structure of a compound's components is crucial for deciphering its meaning. Earlier approaches in Sanskrit have focused on binary compounds and neglected the multi-component compound setting. This work introduces the novel task of nested compound type identification (NeCTI), which aims to identify ne…
▽ More
Multi-component compounding is a prevalent phenomenon in Sanskrit, and understanding the implicit structure of a compound's components is crucial for deciphering its meaning. Earlier approaches in Sanskrit have focused on binary compounds and neglected the multi-component compound setting. This work introduces the novel task of nested compound type identification (NeCTI), which aims to identify nested spans of a multi-component compound and decode the implicit semantic relations between them. To the best of our knowledge, this is the first attempt in the field of lexical semantics to propose this task.
We present 2 newly annotated datasets including an out-of-domain dataset for this task. We also benchmark these datasets by exploring the efficacy of the standard problem formulations such as nested named entity recognition, constituency parsing and seq2seq, etc. We present a novel framework named DepNeCTI: Dependency-based Nested Compound Type Identifier that surpasses the performance of the best baseline with an average absolute improvement of 13.1 points F1-score in terms of Labeled Span Score (LSS) and a 5-fold enhancement in inference efficiency. In line with the previous findings in the binary Sanskrit compound identification task, context provides benefits for the NeCTI task. The codebase and datasets are publicly available at: https://github.com/yaswanth-iitkgp/DepNeCTI
△ Less
Submitted 14 October, 2023;
originally announced October 2023.
-
Modeling interdisciplinary interactions among Physics, Mathematics & Computer Science
Authors:
Rima Hazra,
Mayank Singh,
Pawan Goyal,
Bibhas Adhikari,
Animesh Mukherjee
Abstract:
Interdisciplinarity has over the recent years have gained tremendous importance and has become one of the key ways of doing cutting edge research. In this paper we attempt to model the citation flow across three different fields -- Physics (PHY), Mathematics (MA) and Computer Science (CS). For instance, is there a specific pattern in which these fields cite one another? We carry out experiments on…
▽ More
Interdisciplinarity has over the recent years have gained tremendous importance and has become one of the key ways of doing cutting edge research. In this paper we attempt to model the citation flow across three different fields -- Physics (PHY), Mathematics (MA) and Computer Science (CS). For instance, is there a specific pattern in which these fields cite one another? We carry out experiments on a dataset comprising more than 1.2 million articles taken from these three fields. We quantify the citation interactions among these three fields through temporal bucket signatures. We present numerical models based on variants of the recently proposed relay-linking framework to explain the citation dynamics across the three disciplines. These models make a modest attempt to unfold the underlying principles of how citation links could have been formed across the three fields over time.
△ Less
Submitted 19 September, 2023;
originally announced September 2023.
-
A Robust SINDy Approach by Combining Neural Networks and an Integral Form
Authors:
Ali Forootani,
Pawan Goyal,
Peter Benner
Abstract:
The discovery of governing equations from data has been an active field of research for decades. One widely used methodology for this purpose is sparse regression for nonlinear dynamics, known as SINDy. Despite several attempts, noisy and scarce data still pose a severe challenge to the success of the SINDy approach. In this work, we discuss a robust method to discover nonlinear governing equation…
▽ More
The discovery of governing equations from data has been an active field of research for decades. One widely used methodology for this purpose is sparse regression for nonlinear dynamics, known as SINDy. Despite several attempts, noisy and scarce data still pose a severe challenge to the success of the SINDy approach. In this work, we discuss a robust method to discover nonlinear governing equations from noisy and scarce data. To do this, we make use of neural networks to learn an implicit representation based on measurement data so that not only it produces the output in the vicinity of the measurements but also the time-evolution of output can be described by a dynamical system. Additionally, we learn such a dynamic system in the spirit of the SINDy framework. Leveraging the implicit representation using neural networks, we obtain the derivative information -- required for SINDy -- using an automatic differentiation tool. To enhance the robustness of our methodology, we further incorporate an integral condition on the output of the implicit networks. Furthermore, we extend our methodology to handle data collected from multiple initial conditions. We demonstrate the efficiency of the proposed methodology to discover governing equations under noisy and scarce data regimes by means of several examples and compare its performance with existing methods.
△ Less
Submitted 13 September, 2023;
originally announced September 2023.
-
Deep Learning for Structure-Preserving Universal Stable Koopman-Inspired Embeddings for Nonlinear Canonical Hamiltonian Dynamics
Authors:
Pawan Goyal,
Süleyman Yıldız,
Peter Benner
Abstract:
Discovering a suitable coordinate transformation for nonlinear systems enables the construction of simpler models, facilitating prediction, control, and optimization for complex nonlinear systems. To that end, Koopman operator theory offers a framework for global linearization for nonlinear systems, thereby allowing the usage of linear tools for design studies. In this work, we focus on the identi…
▽ More
Discovering a suitable coordinate transformation for nonlinear systems enables the construction of simpler models, facilitating prediction, control, and optimization for complex nonlinear systems. To that end, Koopman operator theory offers a framework for global linearization for nonlinear systems, thereby allowing the usage of linear tools for design studies. In this work, we focus on the identification of global linearized embeddings for canonical nonlinear Hamiltonian systems through a symplectic transformation. While this task is often challenging, we leverage the power of deep learning to discover the desired embeddings. Furthermore, to overcome the shortcomings of Koopman operators for systems with continuous spectra, we apply the lifting principle and learn global cubicized embeddings. Additionally, a key emphasis is paid to enforce the bounded stability for the dynamics of the discovered embeddings. We demonstrate the capabilities of deep learning in acquiring compact symplectic coordinate transformation and the corresponding simple dynamical models, fostering data-driven learning of nonlinear canonical Hamiltonian systems, even those with continuous spectra.
△ Less
Submitted 26 August, 2023;
originally announced August 2023.
-
Guaranteed Stable Quadratic Models and their applications in SINDy and Operator Inference
Authors:
Pawan Goyal,
Igor Pontes Duff,
Peter Benner
Abstract:
Scientific machine learning for inferring dynamical systems combines data-driven modeling, physics-based modeling, and empirical knowledge. It plays an essential role in engineering design and digital twinning. In this work, we primarily focus on an operator inference methodology that builds dynamical models, preferably in low-dimension, with a prior hypothesis on the model structure, often determ…
▽ More
Scientific machine learning for inferring dynamical systems combines data-driven modeling, physics-based modeling, and empirical knowledge. It plays an essential role in engineering design and digital twinning. In this work, we primarily focus on an operator inference methodology that builds dynamical models, preferably in low-dimension, with a prior hypothesis on the model structure, often determined by known physics or given by experts. Then, for inference, we aim to learn the operators of a model by setting up an appropriate optimization problem. One of the critical properties of dynamical systems is stability. However, this property is not guaranteed by the inferred models. In this work, we propose inference formulations to learn quadratic models, which are stable by design. Precisely, we discuss the parameterization of quadratic systems that are locally and globally stable. Moreover, for quadratic systems with no stable point yet bounded (e.g., chaotic Lorenz model), we discuss how to parameterize such bounded behaviors in the learning process. Using those parameterizations, we set up inference problems, which are then solved using a gradient-based optimization method. Furthermore, to avoid numerical derivatives and still learn continuous systems, we make use of an integral form of differential equations. We present several numerical examples, illustrating the preservation of stability and discussing its comparison with the existing state-of-the-art approach to infer operators. By means of numerical examples, we also demonstrate how the proposed methods are employed to discover governing equations and energy-preserving models.
△ Less
Submitted 7 January, 2024; v1 submitted 26 August, 2023;
originally announced August 2023.
-
Aesthetics of Sanskrit Poetry from the Perspective of Computational Linguistics: A Case Study Analysis on Siksastaka
Authors:
Jivnesh Sandhan,
Amruta Barbadikar,
Malay Maity,
Pavankumar Satuluri,
Tushar Sandhan,
Ravi M. Gupta,
Pawan Goyal,
Laxmidhar Behera
Abstract:
Sanskrit poetry has played a significant role in shaping the literary and cultural landscape of the Indian subcontinent for centuries. However, not much attention has been devoted to uncovering the hidden beauty of Sanskrit poetry in computational linguistics. This article explores the intersection of Sanskrit poetry and computational linguistics by proposing a roadmap of an interpretable framewor…
▽ More
Sanskrit poetry has played a significant role in shaping the literary and cultural landscape of the Indian subcontinent for centuries. However, not much attention has been devoted to uncovering the hidden beauty of Sanskrit poetry in computational linguistics. This article explores the intersection of Sanskrit poetry and computational linguistics by proposing a roadmap of an interpretable framework to analyze and classify the qualities and characteristics of fine Sanskrit poetry. We discuss the rich tradition of Sanskrit poetry and the significance of computational linguistics in automatically identifying the characteristics of fine poetry. The proposed framework involves a human-in-the-loop approach that combines deterministic aspects delegated to machines and deep semantics left to human experts. We provide a deep analysis of Siksastaka, a Sanskrit poem, from the perspective of 6 prominent kavyashastra schools, to illustrate the proposed framework. Additionally, we provide compound, dependency, anvaya (prose order linearised form), meter, rasa (mood), alankar (figure of speech), and riti (writing style) annotations for Siksastaka and a web application to illustrate the poem's analysis and annotations. Our key contributions include the proposed framework, the analysis of Siksastaka, the annotations and the web application for future research. Link for interactive analysis: https://sanskritshala.github.io/shikshastakam/
△ Less
Submitted 14 August, 2023;
originally announced August 2023.
-
Alexa, play with robot: Introducing the First Alexa Prize SimBot Challenge on Embodied AI
Authors:
Hangjie Shi,
Leslie Ball,
Govind Thattai,
Desheng Zhang,
Lucy Hu,
Qiaozi Gao,
Suhaila Shakiah,
Xiaofeng Gao,
Aishwarya Padmakumar,
Bofei Yang,
Cadence Chung,
Dinakar Guthy,
Gaurav Sukhatme,
Karthika Arumugam,
Matthew Wen,
Osman Ipek,
Patrick Lange,
Rohan Khanna,
Shreyas Pansare,
Vasu Sharma,
Chao Zhang,
Cris Flagg,
Daniel Pressel,
Lavina Vaz,
Luke Dai
, et al. (17 additional authors not shown)
Abstract:
The Alexa Prize program has empowered numerous university students to explore, experiment, and showcase their talents in building conversational agents through challenges like the SocialBot Grand Challenge and the TaskBot Challenge. As conversational agents increasingly appear in multimodal and embodied contexts, it is important to explore the affordances of conversational interaction augmented wi…
▽ More
The Alexa Prize program has empowered numerous university students to explore, experiment, and showcase their talents in building conversational agents through challenges like the SocialBot Grand Challenge and the TaskBot Challenge. As conversational agents increasingly appear in multimodal and embodied contexts, it is important to explore the affordances of conversational interaction augmented with computer vision and physical embodiment. This paper describes the SimBot Challenge, a new challenge in which university teams compete to build robot assistants that complete tasks in a simulated physical environment. This paper provides an overview of the SimBot Challenge, which included both online and offline challenge phases. We describe the infrastructure and support provided to the teams including Alexa Arena, the simulated environment, and the ML toolkit provided to teams to accelerate their building of vision and language models. We summarize the approaches the participating teams took to overcome research challenges and extract key lessons learned. Finally, we provide analysis of the performance of the competing SimBots during the competition.
△ Less
Submitted 9 August, 2023;
originally announced August 2023.
-
FLIRT: Feedback Loop In-context Red Teaming
Authors:
Ninareh Mehrabi,
Palash Goyal,
Christophe Dupuy,
Qian Hu,
Shalini Ghosh,
Richard Zemel,
Kai-Wei Chang,
Aram Galstyan,
Rahul Gupta
Abstract:
Warning: this paper contains content that may be inappropriate or offensive.
As generative models become available for public use in various applications, testing and analyzing vulnerabilities of these models has become a priority. Here we propose an automatic red teaming framework that evaluates a given model and exposes its vulnerabilities against unsafe and inappropriate content generation. O…
▽ More
Warning: this paper contains content that may be inappropriate or offensive.
As generative models become available for public use in various applications, testing and analyzing vulnerabilities of these models has become a priority. Here we propose an automatic red teaming framework that evaluates a given model and exposes its vulnerabilities against unsafe and inappropriate content generation. Our framework uses in-context learning in a feedback loop to red team models and trigger them into unsafe content generation. We propose different in-context attack strategies to automatically learn effective and diverse adversarial prompts for text-to-image models. Our experiments demonstrate that compared to baseline approaches, our proposed strategy is significantly more effective in exposing vulnerabilities in Stable Diffusion (SD) model, even when the latter is enhanced with safety features. Furthermore, we demonstrate that the proposed framework is effective for red teaming text-to-text models, resulting in significantly higher toxic response generation rate compared to previously reported numbers.
△ Less
Submitted 8 August, 2023;
originally announced August 2023.
-
Parameterized Interpolation of Passive Systems
Authors:
Peter Benner,
Pawan Goyal,
Paul Van Dooren
Abstract:
We study the tangential interpolation problem for a passive transfer function in standard state-space form. We derive new interpolation conditions based on the computation of a deflating subspace associated with a selection of spectral zeros of a parameterized para-Hermitian transfer function. We show that this technique improves the robustness of the low order model and that it can also be applie…
▽ More
We study the tangential interpolation problem for a passive transfer function in standard state-space form. We derive new interpolation conditions based on the computation of a deflating subspace associated with a selection of spectral zeros of a parameterized para-Hermitian transfer function. We show that this technique improves the robustness of the low order model and that it can also be applied to non-passive systems, provided they have sufficiently many spectral zeros in the open right half plane. We analyze the accuracy needed for the computation of the deflating subspace, in order to still have a passive lower order model and we derive a novel selection procedure of spectral zeros in order to obtain low order models with a small approximation error.
△ Less
Submitted 7 August, 2023;
originally announced August 2023.
-
Linearly Implicit Global Energy Preserving Reduced-order Models for Cubic Hamiltonian Systems
Authors:
Süleyman Yildiz,
Pawan Goyal,
Peter Benner
Abstract:
This work discusses the model reduction problem for large-scale multi-symplectic PDEs with cubic invariants. For this, we present a linearly implicit global energy-preserving method to construct reduced-order models. This allows to construct reduced-order models in the form of Hamiltonian systems suitable for long-time integration. Furthermore, We prove that the constructed reduced-order models pr…
▽ More
This work discusses the model reduction problem for large-scale multi-symplectic PDEs with cubic invariants. For this, we present a linearly implicit global energy-preserving method to construct reduced-order models. This allows to construct reduced-order models in the form of Hamiltonian systems suitable for long-time integration. Furthermore, We prove that the constructed reduced-order models preserve global energy, and the spatially discrete equations also preserve the spatially-discrete local energy conversation law. We illustrate the efficiency of the proposed method using three numerical examples, namely a linear wave equation, the Korteweg-de Vries equation, and the Camassa-Holm equation, and present a comparison with the classical POD-Galerkin method.
△ Less
Submitted 4 August, 2023;
originally announced August 2023.
-
Food Classification using Joint Representation of Visual and Textual Data
Authors:
Prateek Mittal,
Puneet Goyal,
Joohi Chauhan
Abstract:
Food classification is an important task in health care. In this work, we propose a multimodal classification framework that uses the modified version of EfficientNet with the Mish activation function for image classification, and the traditional BERT transformer-based network is used for text classification. The proposed network and the other state-of-the-art methods are evaluated on a large open…
▽ More
Food classification is an important task in health care. In this work, we propose a multimodal classification framework that uses the modified version of EfficientNet with the Mish activation function for image classification, and the traditional BERT transformer-based network is used for text classification. The proposed network and the other state-of-the-art methods are evaluated on a large open-source dataset, UPMC Food-101. The experimental results show that the proposed network outperforms the other methods, a significant difference of 11.57% and 6.34% in accuracy is observed for image and text classification, respectively, when compared with the second-best performing method. We also compared the performance in terms of accuracy, precision, and recall for text classification using both machine learning and deep learning-based models. The comparative analysis from the prediction results of both images and text demonstrated the efficiency and robustness of the proposed approach.
△ Less
Submitted 30 August, 2023; v1 submitted 3 August, 2023;
originally announced August 2023.
-
Data-Driven Identification of Quadratic Representations for Nonlinear Hamiltonian Systems using Weakly Symplectic Liftings
Authors:
Süleyman Yildiz,
Pawan Goyal,
Thomas Bendokat,
Peter Benner
Abstract:
We present a framework for learning Hamiltonian systems using data. This work is based on a lifting hypothesis, which posits that nonlinear Hamiltonian systems can be written as nonlinear systems with cubic Hamiltonians. By leveraging this, we obtain quadratic dynamics that are Hamiltonian in a transformed coordinate system. To that end, for given generalized position and momentum data, we propose…
▽ More
We present a framework for learning Hamiltonian systems using data. This work is based on a lifting hypothesis, which posits that nonlinear Hamiltonian systems can be written as nonlinear systems with cubic Hamiltonians. By leveraging this, we obtain quadratic dynamics that are Hamiltonian in a transformed coordinate system. To that end, for given generalized position and momentum data, we propose a methodology to learn quadratic dynamical systems, enforcing the Hamiltonian structure in combination with a weakly-enforced symplectic auto-encoder. The obtained Hamiltonian structure exhibits long-term stability of the system, while the cubic Hamiltonian function provides relatively low model complexity. For low-dimensional data, we determine a higher-dimensional transformed coordinate system, whereas for high-dimensional data, we find a lower-dimensional coordinate system with the desired properties. We demonstrate the proposed methodology by means of both low-dimensional and high-dimensional nonlinear Hamiltonian systems.
△ Less
Submitted 8 February, 2024; v1 submitted 2 August, 2023;
originally announced August 2023.
-
SepHRNet: Generating High-Resolution Crop Maps from Remote Sensing imagery using HRNet with Separable Convolution
Authors:
Priyanka Goyal,
Sohan Patnaik,
Adway Mitra,
Manjira Sinha
Abstract:
The accurate mapping of crop production is crucial for ensuring food security, effective resource management, and sustainable agricultural practices. One way to achieve this is by analyzing high-resolution satellite imagery. Deep Learning has been successful in analyzing images, including remote sensing imagery. However, capturing intricate crop patterns is challenging due to their complexity and…
▽ More
The accurate mapping of crop production is crucial for ensuring food security, effective resource management, and sustainable agricultural practices. One way to achieve this is by analyzing high-resolution satellite imagery. Deep Learning has been successful in analyzing images, including remote sensing imagery. However, capturing intricate crop patterns is challenging due to their complexity and variability. In this paper, we propose a novel Deep learning approach that integrates HRNet with Separable Convolutional layers to capture spatial patterns and Self-attention to capture temporal patterns of the data. The HRNet model acts as a backbone and extracts high-resolution features from crop images. Spatially separable convolution in the shallow layers of the HRNet model captures intricate crop patterns more effectively while reducing the computational cost. The multi-head attention mechanism captures long-term temporal dependencies from the encoded vector representation of the images. Finally, a CNN decoder generates a crop map from the aggregated representation. Adaboost is used on top of this to further improve accuracy. The proposed algorithm achieves a high classification accuracy of 97.5\% and IoU of 55.2\% in generating crop maps. We evaluate the performance of our pipeline on the Zuericrop dataset and demonstrate that our results outperform state-of-the-art models such as U-Net++, ResNet50, VGG19, InceptionV3, DenseNet, and EfficientNet. This research showcases the potential of Deep Learning for Earth Observation Systems.
△ Less
Submitted 11 July, 2023;
originally announced July 2023.
-
CrysMMNet: Multimodal Representation for Crystal Property Prediction
Authors:
Kishalay Das,
Pawan Goyal,
Seung-Cheol Lee,
Satadeep Bhattacharjee,
Niloy Ganguly
Abstract:
Machine Learning models have emerged as a powerful tool for fast and accurate prediction of different crystalline properties. Exiting state-of-the-art models rely on a single modality of crystal data i.e. crystal graph structure, where they construct multi-graph by establishing edges between nearby atoms in 3D space and apply GNN to learn materials representation. Thereby, they encode local chemic…
▽ More
Machine Learning models have emerged as a powerful tool for fast and accurate prediction of different crystalline properties. Exiting state-of-the-art models rely on a single modality of crystal data i.e. crystal graph structure, where they construct multi-graph by establishing edges between nearby atoms in 3D space and apply GNN to learn materials representation. Thereby, they encode local chemical semantics around the atoms successfully but fail to capture important global periodic structural information like space group number, crystal symmetry, rotational information, etc, which influence different crystal properties. In this work, we leverage textual descriptions of materials to model global structural information into graph structure and learn a more robust and enriched representation of crystalline materials. To this effect, we first curate a textual dataset for crystalline material databases containing descriptions of each material. Further, we propose CrysMMNet, a simple multi-modal framework, which fuses both structural and textual representation together to generate a joint multimodal representation of crystalline materials. We conduct extensive experiments on two benchmark datasets across ten different properties to show that CrysMMNet outperforms existing state-of-the-art baseline methods with a good margin. We also observe that fusing the textual representation with crystal graph structure provides consistent improvement for all the SOTA GNN models compared to their own vanilla versions. We have shared the textual dataset, that we have curated for both the benchmark material databases, with the community for future use.
△ Less
Submitted 9 June, 2023;
originally announced July 2023.
-
$FastDoc$: Domain-Specific Fast Pre-training Technique using Document-Level Metadata and Taxonomy
Authors:
Abhilash Nandy,
Manav Nitin Kapadnis,
Sohan Patnaik,
Yash Parag Butala,
Pawan Goyal,
Niloy Ganguly
Abstract:
As the demand for sophisticated Natural Language Processing (NLP) models continues to grow, so does the need for efficient pre-training techniques. Current NLP models undergo resource-intensive pre-training. In response, we introduce $FastDoc$ (Fast Pre-training Technique using Document-Level Metadata and Taxonomy), a novel approach designed to significantly reduce computational demands.…
▽ More
As the demand for sophisticated Natural Language Processing (NLP) models continues to grow, so does the need for efficient pre-training techniques. Current NLP models undergo resource-intensive pre-training. In response, we introduce $FastDoc$ (Fast Pre-training Technique using Document-Level Metadata and Taxonomy), a novel approach designed to significantly reduce computational demands. $FastDoc$ leverages document metadata and domain-specific taxonomy as supervision signals. It involves continual pre-training of an open-domain transformer encoder using sentence-level embeddings, followed by fine-tuning using token-level embeddings. We evaluate $FastDoc$ on six tasks across nine datasets spanning three distinct domains. Remarkably, $FastDoc$ achieves remarkable compute reductions of approximately 1,000x, 4,500x, 500x compared to competitive approaches in Customer Support, Scientific, and Legal domains, respectively. Importantly, these efficiency gains do not compromise performance relative to competitive baselines. Furthermore, reduced pre-training data mitigates catastrophic forgetting, ensuring consistent performance in open-domain scenarios. $FastDoc$ offers a promising solution for resource-efficient pre-training, with potential applications spanning various domains.
△ Less
Submitted 14 November, 2023; v1 submitted 9 June, 2023;
originally announced June 2023.
-
FinRED: A Dataset for Relation Extraction in Financial Domain
Authors:
Soumya Sharma,
Tapas Nayak,
Arusarka Bose,
Ajay Kumar Meena,
Koustuv Dasgupta,
Niloy Ganguly,
Pawan Goyal
Abstract:
Relation extraction models trained on a source domain cannot be applied on a different target domain due to the mismatch between relation sets. In the current literature, there is no extensive open-source relation extraction dataset specific to the finance domain. In this paper, we release FinRED, a relation extraction dataset curated from financial news and earning call transcripts containing rel…
▽ More
Relation extraction models trained on a source domain cannot be applied on a different target domain due to the mismatch between relation sets. In the current literature, there is no extensive open-source relation extraction dataset specific to the finance domain. In this paper, we release FinRED, a relation extraction dataset curated from financial news and earning call transcripts containing relations from the finance domain. FinRED has been created by mapping Wikidata triplets using distance supervision method. We manually annotate the test data to ensure proper evaluation. We also experiment with various state-of-the-art relation extraction models on this dataset to create the benchmark. We see a significant drop in their performance on FinRED compared to the general relation extraction datasets which tells that we need better models for financial relation extraction.
△ Less
Submitted 6 June, 2023;
originally announced June 2023.
-
Financial Numeric Extreme Labelling: A Dataset and Benchmarking for XBRL Tagging
Authors:
Soumya Sharma,
Subhendu Khatuya,
Manjunath Hegde,
Afreen Shaikh. Koustuv Dasgupta,
Pawan Goyal,
Niloy Ganguly
Abstract:
The U.S. Securities and Exchange Commission (SEC) mandates all public companies to file periodic financial statements that should contain numerals annotated with a particular label from a taxonomy. In this paper, we formulate the task of automating the assignment of a label to a particular numeral span in a sentence from an extremely large label set. Towards this task, we release a dataset, Financ…
▽ More
The U.S. Securities and Exchange Commission (SEC) mandates all public companies to file periodic financial statements that should contain numerals annotated with a particular label from a taxonomy. In this paper, we formulate the task of automating the assignment of a label to a particular numeral span in a sentence from an extremely large label set. Towards this task, we release a dataset, Financial Numeric Extreme Labelling (FNXL), annotated with 2,794 labels. We benchmark the performance of the FNXL dataset by formulating the task as (a) a sequence labelling problem and (b) a pipeline with span extraction followed by Extreme Classification. Although the two approaches perform comparably, the pipeline solution provides a slight edge for the least frequent labels.
△ Less
Submitted 6 June, 2023;
originally announced June 2023.