-
The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models
Authors:
Seungone Kim,
Juyoung Suk,
Ji Yong Cho,
Shayne Longpre,
Chaeeun Kim,
Dongkeun Yoon,
Guijin Son,
Yejin Cho,
Sheikh Shafayat,
Jinheon Baek,
Sue Hyun Park,
Hyeonbin Hwang,
Jinkyung Jo,
Hyowon Cho,
Haebin Shin,
Seongyun Lee,
Hanseok Oh,
Noah Lee,
Namgyu Ho,
Se June Joo,
Miyoung Ko,
Yoonjoo Lee,
Hyungjoo Chae,
Jamin Shin,
Joel Jang
, et al. (7 additional authors not shown)
Abstract:
As language models (LMs) become capable of handling a wide range of tasks, their evaluation is becoming as challenging as their development. Most generation benchmarks currently assess LMs using abstract evaluation criteria like helpfulness and harmlessness, which often lack the flexibility and granularity of human assessment. Additionally, these benchmarks tend to focus disproportionately on spec…
▽ More
As language models (LMs) become capable of handling a wide range of tasks, their evaluation is becoming as challenging as their development. Most generation benchmarks currently assess LMs using abstract evaluation criteria like helpfulness and harmlessness, which often lack the flexibility and granularity of human assessment. Additionally, these benchmarks tend to focus disproportionately on specific capabilities such as instruction following, leading to coverage bias. To overcome these limitations, we introduce the BiGGen Bench, a principled generation benchmark designed to thoroughly evaluate nine distinct capabilities of LMs across 77 diverse tasks. A key feature of the BiGGen Bench is its use of instance-specific evaluation criteria, closely mirroring the nuanced discernment of human evaluation. We apply this benchmark to assess 103 frontier LMs using five evaluator LMs. Our code, data, and evaluation results are all publicly available at https://github.com/prometheus-eval/prometheus-eval/tree/main/BiGGen-Bench.
△ Less
Submitted 9 June, 2024;
originally announced June 2024.
-
The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning
Authors:
Seungone Kim,
Se June Joo,
Doyoung Kim,
Joel Jang,
Seonghyeon Ye,
Jamin Shin,
Minjoon Seo
Abstract:
Language models (LMs) with less than 100B parameters are known to perform poorly on chain-of-thought (CoT) reasoning in contrast to large LMs when solving unseen tasks. In this work, we aim to equip smaller LMs with the step-by-step reasoning capability by instruction tuning with CoT rationales. In order to achieve this goal, we first introduce a new instruction-tuning dataset called the CoT Colle…
▽ More
Language models (LMs) with less than 100B parameters are known to perform poorly on chain-of-thought (CoT) reasoning in contrast to large LMs when solving unseen tasks. In this work, we aim to equip smaller LMs with the step-by-step reasoning capability by instruction tuning with CoT rationales. In order to achieve this goal, we first introduce a new instruction-tuning dataset called the CoT Collection, which augments the existing Flan Collection (including only 9 CoT tasks) with additional 1.84 million rationales across 1,060 tasks. We show that CoT fine-tuning Flan-T5 (3B & 11B) with CoT Collection enables smaller LMs to have better CoT capabilities on unseen tasks. On the BIG-Bench-Hard (BBH) benchmark, we report an average improvement of +4.34% (Flan-T5 3B) and +2.60% (Flan-T5 11B), in terms of zero-shot task accuracy. Furthermore, we show that instruction tuning with CoT Collection allows LMs to possess stronger few-shot learning capabilities on 4 domain-specific tasks, resulting in an improvement of +2.24% (Flan-T5 3B) and +2.37% (Flan-T5 11B), even outperforming ChatGPT utilizing demonstrations until the max length by a +13.98% margin. Our code, the CoT Collection data, and model checkpoints are publicly available.
△ Less
Submitted 14 October, 2023; v1 submitted 23 May, 2023;
originally announced May 2023.
-
CoTEVer: Chain of Thought Prompting Annotation Toolkit for Explanation Verification
Authors:
Seungone Kim,
Se June Joo,
Yul Jang,
Hyungjoo Chae,
Jinyoung Yeo
Abstract:
Chain-of-thought (CoT) prompting enables large language models (LLMs) to solve complex reasoning tasks by generating an explanation before the final prediction. Despite it's promising ability, a critical downside of CoT prompting is that the performance is greatly affected by the factuality of the generated explanation. To improve the correctness of the explanations, fine-tuning language models wi…
▽ More
Chain-of-thought (CoT) prompting enables large language models (LLMs) to solve complex reasoning tasks by generating an explanation before the final prediction. Despite it's promising ability, a critical downside of CoT prompting is that the performance is greatly affected by the factuality of the generated explanation. To improve the correctness of the explanations, fine-tuning language models with explanation data is needed. However, there exists only a few datasets that can be used for such approaches, and no data collection tool for building them. Thus, we introduce CoTEVer, a tool-kit for annotating the factual correctness of generated explanations and collecting revision data of wrong explanations. Furthermore, we suggest several use cases where the data collected with CoTEVer can be utilized for enhancing the faithfulness of explanations. Our toolkit is publicly available at https://github.com/SeungoneKim/CoTEVer.
△ Less
Submitted 6 March, 2023;
originally announced March 2023.
-
Mind the Gap! Injecting Commonsense Knowledge for Abstractive Dialogue Summarization
Authors:
Seungone Kim,
Se June Joo,
Hyungjoo Chae,
Chaehyeong Kim,
Seung-won Hwang,
Jinyoung Yeo
Abstract:
In this paper, we propose to leverage the unique characteristics of dialogues sharing commonsense knowledge across participants, to resolve the difficulties in summarizing them. We present SICK, a framework that uses commonsense inferences as additional context. Compared to previous work that solely relies on the input dialogue, SICK uses an external knowledge model to generate a rich set of commo…
▽ More
In this paper, we propose to leverage the unique characteristics of dialogues sharing commonsense knowledge across participants, to resolve the difficulties in summarizing them. We present SICK, a framework that uses commonsense inferences as additional context. Compared to previous work that solely relies on the input dialogue, SICK uses an external knowledge model to generate a rich set of commonsense inferences and selects the most probable one with a similarity-based selection method. Built upon SICK, SICK++ utilizes commonsense as supervision, where the task of generating commonsense inferences is added upon summarizing the dialogue in a multi-task learning setting. Experimental results show that with injected commonsense knowledge, our framework generates more informative and consistent summaries than existing methods.
△ Less
Submitted 2 September, 2022;
originally announced September 2022.
-
Tunneling anisotropic magnetoresistance in multilayer-(Co/Pt)/AlOx/Pt structures
Authors:
B. G. Park,
J. Wunderlich,
D. A. Williams,
S. J. Joo,
K. Y. Jung,
K. H. Shin,
K. Olejnik,
A. B. Shick,
T. Jungwirth
Abstract:
We report observations of tunneling anisotropic magnetoresitance (TAMR) in vertical tunnel devices with a ferromagnetic multilayer-(Co/Pt) electrode and a non-magnetic Pt counter-electrode separated by an AlOx barrier. In stacks with the ferromagnetic electrode terminated by a Co film the TAMR magnitude saturates at 0.15% beyond which it shows only weak dependence on the magnetic field strength,…
▽ More
We report observations of tunneling anisotropic magnetoresitance (TAMR) in vertical tunnel devices with a ferromagnetic multilayer-(Co/Pt) electrode and a non-magnetic Pt counter-electrode separated by an AlOx barrier. In stacks with the ferromagnetic electrode terminated by a Co film the TAMR magnitude saturates at 0.15% beyond which it shows only weak dependence on the magnetic field strength, bias voltage, and temperature. For ferromagnetic electrodes terminated by two monolayers of Pt we observe order(s) of magnitude enhancement of the TAMR and a strong dependence on field, temperature and bias. Discussion of experiments is based on relativistic ab initio calculations of magnetization orientation dependent densities of states of Co and Co/Pt model systems.
△ Less
Submitted 8 January, 2008;
originally announced January 2008.