Violence-provoking speech – speech that implicitly or explicitly promotes violence against the members of the targeted community, contributed to a massive surge in anti-Asian crimes during the COVID-19 pandemic. While previous works have characterized and built tools for detecting other forms of harmful speech, like fear speech and hate speech, our work takes a community-centric approach to studying anti-Asian violence-provoking speech. Using data from ~420k Twitter posts spanning a 3-year duration (January 1, 2020 to February 1, 2023), we develop a codebook to characterize anti-Asian violence-provoking speech and collect a community-crowdsourced dataset to facilitate its large-scale detection using state-of-the-art classifiers. We contrast the capabilities of natural language processing classifiers, ranging from BERT-based to LLM-based classifiers, in detecting violence-provoking speech with their capabilities to detect anti-Asian hateful speech. In contrast to prior work that has demonstrated the effectiveness of such classifiers in detecting hateful speech (F1 = 0.89), our work shows that accurate and reliable detection of violence-provoking speech is a challenging task (F1 = 0.69). We discuss the implications of our findings, particularly the need for proactive interventions to support Asian communities during public health crises.
The field of unsupervised machine translation has seen significant advancement from the marriage of the Transformer and the back-translation algorithm. The Transformer is a powerful generative model, and back-translation leverages Transformer’s high-quality translations for iterative self-improvement. However, the Transformer is encumbered by the run-time of autoregressive inference during back-translation, and back-translation is limited by a lack of synthetic data efficiency. We propose a two-for-one improvement to Transformer back-translation: Quick Back-Translation (QBT). QBT re-purposes the encoder as a generative model, and uses encoder-generated sequences to train the decoder in conjunction with the original autoregressive back-translation step, improving data throughput and utilization. Experiments on various WMT benchmarks demonstrate that a relatively small number of refining steps of QBT improve current unsupervised machine translation models, and that QBT dramatically outperforms standard back-translation only method in terms of training efficiency for comparable translation qualities.
Transition-based parsers for Abstract Meaning Representation (AMR) rely on node-to-word alignments. These alignments are learned separately from parser training and require a complex pipeline of rule-based components, pre-processing, and post-processing to satisfy domain-specific constraints. Parsers also train on a point-estimate of the alignment pipeline, neglecting the uncertainty due to the inherent ambiguity of alignment. In this work we explore two avenues for overcoming these limitations. First, we propose a neural aligner for AMR that learns node-to-word alignments without relying on complex pipelines. We subsequently explore a tighter integration of aligner and parser training by considering a distribution over oracle action sequences arising from aligner uncertainty. Empirical results show this approach leads to more accurate alignments and generalization better from the AMR2.0 to AMR3.0 corpora. We attain a new state-of-the art for gold-only trained models, matching silver-trained performance without the need for beam search on AMR3.0.
pdf
bib
abs GEMv2: Multilingual NLG Benchmarking in a Single Line of Code Sebastian Gehrmann
|
Abhik Bhattacharjee
|
Abinaya Mahendiran
|
Alex Wang
|
Alexandros Papangelis
|
Aman Madaan
|
Angelina Mcmillan-major
|
Anna Shvets
|
Ashish Upadhyay
|
Bernd Bohnet
|
Bingsheng Yao
|
Bryan Wilie
|
Chandra Bhagavatula
|
Chaobin You
|
Craig Thomson
|
Cristina Garbacea
|
Dakuo Wang
|
Daniel Deutsch
|
Deyi Xiong
|
Di Jin
|
Dimitra Gkatzia
|
Dragomir Radev
|
Elizabeth Clark
|
Esin Durmus
|
Faisal Ladhak
|
Filip Ginter
|
Genta Indra Winata
|
Hendrik Strobelt
|
Hiroaki Hayashi
|
Jekaterina Novikova
|
Jenna Kanerva
|
Jenny Chim
|
Jiawei Zhou
|
Jordan Clive
|
Joshua Maynez
|
João Sedoc
|
Juraj Juraska
|
Kaustubh Dhole
|
Khyathi Raghavi Chandu
|
Laura Perez Beltrachini
|
Leonardo F . R. Ribeiro
|
Lewis Tunstall
|
Li Zhang
|
Mahim Pushkarna
|
Mathias Creutz
|
Michael White
|
Mihir Sanjay Kale
|
Moussa Kamal Eddine
|
Nico Daheim
|
Nishant Subramani
|
Ondrej Dusek
|
Paul Pu Liang
|
Pawan Sasanka Ammanamanchi
|
Qi Zhu
|
Ratish Puduppully
|
Reno Kriz
|
Rifat Shahriyar
|
Ronald Cardenas
|
Saad Mahamood
|
Salomey Osei
|
Samuel Cahyawijaya
|
Sanja Štajner
|
Sebastien Montella
|
Shailza Jolly
|
Simon Mille
|
Tahmid Hasan
|
Tianhao Shen
|
Tosin Adewumi
|
Vikas Raunak
|
Vipul Raheja
|
Vitaly Nikolaev
|
Vivian Tsai
|
Yacine Jernite
|
Ying Xu
|
Yisi Sang
|
Yixin Liu
|
Yufang Hou Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations
Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. The compatibility, often facilitated through leaderboards, thus leads to outdated but standardized evaluation practices. We pose that the standardization is taking place in the wrong spot. Evaluation infrastructure should enable researchers to use the latest methods and what should be standardized instead is how to incorporate these new evaluation advances. We introduce GEMv2, the new version of the Generation, Evaluation, and Metrics Benchmark which uses a modular infrastructure for dataset, model, and metric developers to benefit from each other’s work. GEMv2 supports 40 documented datasets in 51 languages, ongoing online evaluation for all datasets, and our interactive tools make it easier to add new datasets to the living benchmark.
Standard conversational semantic parsing maps a complete user utterance into an executable program, after which the program is executed to respond to the user. This could be slow when the program contains expensive function calls. We investigate the opportunity to reduce latency by predicting and executing function calls while the user is still speaking. We introduce the task of online semantic parsing for this purpose, with a formal latency reduction metric inspired by simultaneous machine translation. We propose a general framework with first a learned prefix-to-program prediction module, and then a simple yet effective thresholding heuristic for subprogram selection for early execution. Experiments on the SMCalFlow and TreeDST datasets show our approach achieves large latency reduction with good parsing quality, with a 30%–65% latency reduction depending on function execution time and allowed cost.
To alleviate the data scarcity problem in training question answering systems, recent works propose additional intermediate pre-training for dense passage retrieval (DPR). However, there still remains a large discrepancy between the provided upstream signals and the downstream question-passage relevance, which leads to less improvement. To bridge this gap, we propose the HyperLink-induced Pre-training (HLP), a method to pre-train the dense retriever with the text relevance induced by hyperlink-based topology within Web documents. We demonstrate that the hyperlink-based structures of dual-link and co-mention can provide effective relevance signals for large-scale pre-training that better facilitate downstream passage retrieval. We investigate the effectiveness of our approach across a wide range of open-domain QA datasets under zero-shot, few-shot, multi-hop, and out-of-domain scenarios. The experiments show our HLP outperforms the BM25 by up to 7 points as well as other pre-training methods by more than 10 points in terms of top-20 retrieval accuracy under the zero-shot scenario. Furthermore, HLP significantly outperforms other pre-training methods under the other scenarios.
2021
pdf
bib
abs The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics Sebastian Gehrmann
|
Tosin Adewumi
|
Karmanya Aggarwal
|
Pawan Sasanka Ammanamanchi
|
Anuoluwapo Aremu
|
Antoine Bosselut
|
Khyathi Raghavi Chandu
|
Miruna-Adriana Clinciu
|
Dipanjan Das
|
Kaustubh Dhole
|
Wanyu Du
|
Esin Durmus
|
Ondřej Dušek
|
Chris Chinenye Emezue
|
Varun Gangal
|
Cristina Garbacea
|
Tatsunori Hashimoto
|
Yufang Hou
|
Yacine Jernite
|
Harsh Jhamtani
|
Yangfeng Ji
|
Shailza Jolly
|
Mihir Kale
|
Dhruv Kumar
|
Faisal Ladhak
|
Aman Madaan
|
Mounica Maddela
|
Khyati Mahajan
|
Saad Mahamood
|
Bodhisattwa Prasad Majumder
|
Pedro Henrique Martins
|
Angelina McMillan-Major
|
Simon Mille
|
Emiel van Miltenburg
|
Moin Nadeem
|
Shashi Narayan
|
Vitaly Nikolaev
|
Andre Niyongabo Rubungo
|
Salomey Osei
|
Ankur Parikh
|
Laura Perez-Beltrachini
|
Niranjan Ramesh Rao
|
Vikas Raunak
|
Juan Diego Rodriguez
|
Sashank Santhanam
|
João Sedoc
|
Thibault Sellam
|
Samira Shaikh
|
Anastasia Shimorina
|
Marco Antonio Sobrevilla Cabezudo
|
Hendrik Strobelt
|
Nishant Subramani
|
Wei Xu
|
Diyi Yang
|
Akhila Yerukola
|
Jiawei Zhou Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)
We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of automated metrics, datasets, and human evaluation standards. Due to this moving target, new models often still evaluate on divergent anglo-centric corpora with well-established, but flawed, metrics. This disconnect makes it challenging to identify the limitations of current models and opportunities for progress. Addressing this limitation, GEM provides an environment in which models can easily be applied to a wide set of tasks and in which evaluation strategies can be tested. Regular updates to the benchmark will help NLG research become more multilingual and evolve the challenge alongside models. This paper serves as the description of the data for the 2021 shared task at the associated GEM Workshop.
Abstract Meaning Representation parsing is a sentence-to-graph prediction task where target nodes are not explicitly aligned to sentence tokens. However, since graph nodes are semantically based on one or more sentence tokens, implicit alignments can be derived. Transition-based parsers operate over the sentence from left to right, capturing this inductive bias via alignments at the cost of limited expressiveness. In this work, we propose a transition-based system that combines hard-attention over sentences with a target-side action pointer mechanism to decouple source tokens from node representations and address alignments. We model the transitions as well as the pointer mechanism through straightforward modifications within a single Transformer architecture. Parser state and graph structure information are efficiently encoded using attention heads. We show that our action-pointer approach leads to increased expressiveness and attains large gains (+1.6 points) against the best transition-based AMR parser in very similar conditions. While using no graph re-categorization, our single model yields the second best Smatch score on AMR 2.0 (81.8), which is further improved to 83.4 with silver data and ensemble decoding.
Predicting linearized Abstract Meaning Representation (AMR) graphs using pre-trained sequence-to-sequence Transformer models has recently led to large improvements on AMR parsing benchmarks. These parsers are simple and avoid explicit modeling of structure but lack desirable properties such as graph well-formedness guarantees or built-in graph-sentence alignments. In this work we explore the integration of general pre-trained sequence-to-sequence language models and a structure-aware transition-based approach. We depart from a pointer-based transition system and propose a simplified transition set, designed to better exploit pre-trained language models for structured fine-tuning. We also explore modeling the parser state within the pre-trained encoder-decoder architecture and different vocabulary strategies for the same purpose. We provide a detailed comparison with recent progress in AMR parsing and show that the proposed parser retains the desirable properties of previous transition-based approaches, while being simpler and reaching the new parsing state of the art for AMR 2.0, without the need for graph re-categorization.
Non-autoregressive (NAR) neural machine translation is usually done via knowledge distillation from an autoregressive (AR) model. Under this framework, we leverage large monolingual corpora to improve the NAR model’s performance, with the goal of transferring the AR model’s generalization ability while preventing overfitting. On top of a strong NAR baseline, our experimental results on the WMT14 En-De and WMT16 En-Ro news translation tasks confirm that monolingual data augmentation consistently improves the performance of the NAR model to approach the teacher AR model’s performance, yields comparable or better results than the best non-iterative NAR methods in the literature and helps reduce overfitting in the training process.
We propose an unsupervised method for sentence summarization using only language modeling. The approach employs two language models, one that is generic (i.e. pretrained), and the other that is specific to the target domain. We show that by using a product-of-experts criteria these are enough for maintaining continuous contextual matching while maintaining output fluency. Experiments on both abstractive and extractive sentence summarization data sets show promising results of our method without being exposed to any paired data.