2024
pdf
bib
abs
More Samples or More Prompts? Exploring Effective Few-Shot In-Context Learning for LLMs with In-Context Sampling
Bingsheng Yao
|
Guiming Chen
|
Ruishi Zou
|
Yuxuan Lu
|
Jiachen Li
|
Shao Zhang
|
Yisi Sang
|
Sijia Liu
|
James Hendler
|
Dakuo Wang
Findings of the Association for Computational Linguistics: NAACL 2024
While most existing works on LLM prompting techniques focus only on how to select a better set of data samples inside one single prompt input (In-Context Learning or ICL), why can not we design and leverage multiple prompts together to further improve the LLM’s performance? In this work, we propose In-Context Sampling (ICS), a low-resource LLM prompting technique to produce confident predictions by optimizing the construction of multiple ICL prompt inputs. Extensive experiments with three open-source LLMs (FlanT5-XL, Mistral-7B, and Mixtral-8x7B) on four NLI datasets (e-SNLI, Multi-NLI, ANLI, and Contract-NLI) and one QA dataset (CommonsenseQA) illustrate that ICS can consistently enhance LLMs’ performance. An in-depth evaluation with three data similarity-based ICS strategies suggests that these strategies can further elevate LLM’s performance, which sheds light on a new yet promising future research direction.
2023
pdf
bib
abs
‘Don’t Get Too Technical with Me’: A Discourse Structure-Based Framework for Automatic Science Journalism
Ronald Cardenas
|
Bingsheng Yao
|
Dakuo Wang
|
Yufang Hou
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
Science journalism refers to the task of reporting technical findings of a scientific paper as a less technical news article to the general public audience. We aim to design an automated system to support this real-world task (i.e., automatic science journalism ) by 1) introducing a newly-constructed and real-world dataset (SciTechNews), with tuples of a publicly-available scientific paper, its corresponding news article, and an expert-written short summary snippet; 2) proposing a novel technical framework that integrates a paper’s discourse structure with its metadata to guide generation; and, 3) demonstrating with extensive automatic and human experiments that our model outperforms other baseline methods (e.g. Alpaca and ChatGPT) in elaborating a content plan meaningful for the target audience, simplify the information selected, and produce a coherent final report in a layman’s style.
pdf
bib
abs
Beyond Labels: Empowering Human Annotators with Natural Language Explanations through a Novel Active-Learning Architecture
Bingsheng Yao
|
Ishan Jindal
|
Lucian Popa
|
Yannis Katsis
|
Sayan Ghosh
|
Lihong He
|
Yuxuan Lu
|
Shashank Srivastava
|
Yunyao Li
|
James Hendler
|
Dakuo Wang
Findings of the Association for Computational Linguistics: EMNLP 2023
Real-world domain experts (e.g., doctors) rarely annotate only a decision label in their day-to-day workflow without providing explanations. Yet, existing low-resource learning techniques, such as Active Learning (AL), that aim to support human annotators mostly focus on the label while neglecting the natural language explanation of a data point. This work proposes a novel AL architecture to support experts’ real-world need for label and explanation annotations in low-resource scenarios. Our AL architecture leverages an explanation-generation model to produce explanations guided by human explanations, a prediction model that utilizes generated explanations toward prediction faithfully, and a novel data diversity-based AL sampling strategy that benefits from the explanation annotations. Automated and human evaluations demonstrate the effectiveness of incorporating explanations into AL sampling and the improved human annotation efficiency and trustworthiness with our AL architecture. Additional ablation studies illustrate the potential of our AL architecture for transfer learning, generalizability, and integration with large language models (LLMs). While LLMs exhibit exceptional explanation-generation capabilities for relatively simple tasks, their effectiveness in complex real-world tasks warrants further in-depth study.
pdf
bib
abs
Are Human Explanations Always Helpful? Towards Objective Evaluation of Human Natural Language Explanations
Bingsheng Yao
|
Prithviraj Sen
|
Lucian Popa
|
James Hendler
|
Dakuo Wang
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Human-annotated labels and explanations are critical for training explainable NLP models. However, unlike human-annotated labels whose quality is easier to calibrate (e.g., with a majority vote), human-crafted free-form explanations can be quite subjective. Before blindly using them as ground truth to train ML models, a vital question needs to be asked: How do we evaluate a human-annotated explanation’s quality? In this paper, we build on the view that the quality of a human-annotated explanation can be measured based on its helpfulness (or impairment) to the ML models’ performance for the desired NLP tasks for which the annotations were collected. In comparison to the commonly used Simulatability score, we define a new metric that can take into consideration the helpfulness of an explanation for model performance at both fine-tuning and inference. With the help of a unified dataset format, we evaluated the proposed metric on five datasets (e.g., e-SNLI) against two model architectures (T5 and BART), and the results show that our proposed metric can objectively evaluate the quality of human-annotated explanations, while Simulatability falls short.
2022
pdf
bib
abs
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code
Sebastian Gehrmann
|
Abhik Bhattacharjee
|
Abinaya Mahendiran
|
Alex Wang
|
Alexandros Papangelis
|
Aman Madaan
|
Angelina Mcmillan-major
|
Anna Shvets
|
Ashish Upadhyay
|
Bernd Bohnet
|
Bingsheng Yao
|
Bryan Wilie
|
Chandra Bhagavatula
|
Chaobin You
|
Craig Thomson
|
Cristina Garbacea
|
Dakuo Wang
|
Daniel Deutsch
|
Deyi Xiong
|
Di Jin
|
Dimitra Gkatzia
|
Dragomir Radev
|
Elizabeth Clark
|
Esin Durmus
|
Faisal Ladhak
|
Filip Ginter
|
Genta Indra Winata
|
Hendrik Strobelt
|
Hiroaki Hayashi
|
Jekaterina Novikova
|
Jenna Kanerva
|
Jenny Chim
|
Jiawei Zhou
|
Jordan Clive
|
Joshua Maynez
|
João Sedoc
|
Juraj Juraska
|
Kaustubh Dhole
|
Khyathi Raghavi Chandu
|
Laura Perez Beltrachini
|
Leonardo F . R. Ribeiro
|
Lewis Tunstall
|
Li Zhang
|
Mahim Pushkarna
|
Mathias Creutz
|
Michael White
|
Mihir Sanjay Kale
|
Moussa Kamal Eddine
|
Nico Daheim
|
Nishant Subramani
|
Ondrej Dusek
|
Paul Pu Liang
|
Pawan Sasanka Ammanamanchi
|
Qi Zhu
|
Ratish Puduppully
|
Reno Kriz
|
Rifat Shahriyar
|
Ronald Cardenas
|
Saad Mahamood
|
Salomey Osei
|
Samuel Cahyawijaya
|
Sanja Štajner
|
Sebastien Montella
|
Shailza Jolly
|
Simon Mille
|
Tahmid Hasan
|
Tianhao Shen
|
Tosin Adewumi
|
Vikas Raunak
|
Vipul Raheja
|
Vitaly Nikolaev
|
Vivian Tsai
|
Yacine Jernite
|
Ying Xu
|
Yisi Sang
|
Yixin Liu
|
Yufang Hou
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations
Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. The compatibility, often facilitated through leaderboards, thus leads to outdated but standardized evaluation practices. We pose that the standardization is taking place in the wrong spot. Evaluation infrastructure should enable researchers to use the latest methods and what should be standardized instead is how to incorporate these new evaluation advances. We introduce GEMv2, the new version of the Generation, Evaluation, and Metrics Benchmark which uses a modular infrastructure for dataset, model, and metric developers to benefit from each other’s work. GEMv2 supports 40 documented datasets in 51 languages, ongoing online evaluation for all datasets, and our interactive tools make it easier to add new datasets to the living benchmark.
pdf
bib
abs
Fantastic Questions and Where to Find Them: FairytaleQA – An Authentic Dataset for Narrative Comprehension
Ying Xu
|
Dakuo Wang
|
Mo Yu
|
Daniel Ritchie
|
Bingsheng Yao
|
Tongshuang Wu
|
Zheng Zhang
|
Toby Li
|
Nora Bradford
|
Branda Sun
|
Tran Hoang
|
Yisi Sang
|
Yufang Hou
|
Xiaojuan Ma
|
Diyi Yang
|
Nanyun Peng
|
Zhou Yu
|
Mark Warschauer
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Question answering (QA) is a fundamental means to facilitate assessment and training of narrative comprehension skills for both machines and young children, yet there is scarcity of high-quality QA datasets carefully designed to serve this purpose. In particular, existing datasets rarely distinguish fine-grained reading skills, such as the understanding of varying narrative elements. Drawing on the reading education research, we introduce FairytaleQA, a dataset focusing on narrative comprehension of kindergarten to eighth-grade students. Generated by educational experts based on an evidence-based theoretical framework, FairytaleQA consists of 10,580 explicit and implicit questions derived from 278 children-friendly stories, covering seven types of narrative elements or relations. Our dataset is valuable in two folds: First, we ran existing QA models on our dataset and confirmed that this annotation helps assess models’ fine-grained learning skills. Second, the dataset supports question generation (QG) task in the education domain. Through benchmarking with QG models, we show that the QG model trained on FairytaleQA is capable of asking high-quality and more diverse questions.
pdf
bib
abs
It is AI’s Turn to Ask Humans a Question: Question-Answer Pair Generation for Children’s Story Books
Bingsheng Yao
|
Dakuo Wang
|
Tongshuang Wu
|
Zheng Zhang
|
Toby Li
|
Mo Yu
|
Ying Xu
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Existing question answering (QA) techniques are created mainly to answer questions asked by humans. But in educational applications, teachers often need to decide what questions they should ask, in order to help students to improve their narrative understanding capabilities. We design an automated question-answer generation (QAG) system for this education scenario: given a story book at the kindergarten to eighth-grade level as input, our system can automatically generate QA pairs that are capable of testing a variety of dimensions of a student’s comprehension skills. Our proposed QAG model architecture is demonstrated using a new expert-annotated FairytaleQA dataset, which has 278 child-friendly storybooks with 10,580 QA pairs. Automatic and human evaluations show that our model outperforms state-of-the-art QAG baseline systems. On top of our QAG system, we also start to build an interactive story-telling application for the future real-world deployment in this educational scenario.
pdf
bib
abs
A Corpus for Commonsense Inference in Story Cloze Test
Bingsheng Yao
|
Ethan Joseph
|
Julian Lioanag
|
Mei Si
Proceedings of the Thirteenth Language Resources and Evaluation Conference
The Story Cloze Test (SCT) is designed for training and evaluating machine learning algorithms for narrative understanding and inferences. The SOTA models can achieve over 90% accuracy on predicting the last sentence. However, it has been shown that high accuracy can be achieved by merely using surface-level features. We suspect these models may not truly understand the story. Based on the SCT dataset, we constructed a human-labeled and human-verified commonsense knowledge inference dataset. Given the first four sentences of a story, we asked crowd-source workers to choose from four types of narrative inference for deciding the ending sentence and which sentence contributes most to the inference. We accumulated data on 1871 stories, and three human workers labeled each story. Analysis of the intra-category and inter-category agreements show a high level of consensus. We present two new tasks for predicting the narrative inference categories and contributing sentences. Our results show that transformer-based models can reach SOTA performance on the original SCT task using transfer learning but don’t perform well on these new and more challenging tasks.
2021
pdf
bib
abs
Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Study
Xiangyang Mou
|
Chenghao Yang
|
Mo Yu
|
Bingsheng Yao
|
Xiaoxiao Guo
|
Saloni Potdar
|
Hui Su
Transactions of the Association for Computational Linguistics, Volume 9
Recent advancements in open-domain question answering (ODQA), that is, finding answers from large open-domain corpus like Wikipedia, have led to human-level performance on many datasets. However, progress in QA over book stories (Book QA) lags despite its similar task formulation to ODQA. This work provides a comprehensive and quantitative analysis about the difficulty of Book QA: (1) We benchmark the research on the NarrativeQA dataset with extensive experiments with cutting-edge ODQA techniques. This quantifies the challenges Book QA poses, as well as advances the published state-of-the-art with a ∼7% absolute improvement on ROUGE-L. (2) We further analyze the detailed challenges in Book QA through human studies.1 Our findings indicate that the event-centric questions dominate this task, which exemplifies the inability of existing QA models to handle event-oriented scenarios.
2020
pdf
bib
abs
Frustratingly Hard Evidence Retrieval for QA Over Books
Xiangyang Mou
|
Mo Yu
|
Bingsheng Yao
|
Chenghao Yang
|
Xiaoxiao Guo
|
Saloni Potdar
|
Hui Su
Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events
A lot of progress has been made to improve question answering (QA) in recent years, but the special problem of QA over narrative book stories has not been explored in-depth. We formulate BookQA as an open-domain QA task given its similar dependency on evidence retrieval. We further investigate how state-of-the-art open-domain QA approaches can help BookQA. Besides achieving state-of-the-art on the NarrativeQA benchmark, our study also reveals the difficulty of evidence retrieval in books with a wealth of experiments and analysis - which necessitates future effort on novel solutions for evidence retrieval in BookQA.