-
Improve Mathematical Reasoning in Language Models by Automated Process Supervision
Authors:
Liangchen Luo,
Yinxiao Liu,
Rosanne Liu,
Samrat Phatale,
Harsh Lara,
Yunxuan Li,
Lei Shu,
Yun Zhu,
Lei Meng,
Jiao Sun,
Abhinav Rastogi
Abstract:
Complex multi-step reasoning tasks, such as solving mathematical problems or generating code, remain a significant hurdle for even the most advanced large language models (LLMs). Verifying LLM outputs with an Outcome Reward Model (ORM) is a standard inference-time technique aimed at enhancing the reasoning performance of LLMs. However, this still proves insufficient for reasoning tasks with a leng…
▽ More
Complex multi-step reasoning tasks, such as solving mathematical problems or generating code, remain a significant hurdle for even the most advanced large language models (LLMs). Verifying LLM outputs with an Outcome Reward Model (ORM) is a standard inference-time technique aimed at enhancing the reasoning performance of LLMs. However, this still proves insufficient for reasoning tasks with a lengthy or multi-hop reasoning chain, where the intermediate outcomes are neither properly rewarded nor penalized. Process supervision addresses this limitation by assigning intermediate rewards during the reasoning process. To date, the methods used to collect process supervision data have relied on either human annotation or per-step Monte Carlo estimation, both prohibitively expensive to scale, thus hindering the broad application of this technique. In response to this challenge, we propose a novel divide-and-conquer style Monte Carlo Tree Search (MCTS) algorithm named \textit{OmegaPRM} for the efficient collection of high-quality process supervision data. This algorithm swiftly identifies the first error in the Chain of Thought (CoT) with binary search and balances the positive and negative examples, thereby ensuring both efficiency and quality. As a result, we are able to collect over 1.5 million process supervision annotations to train a Process Reward Model (PRM). Utilizing this fully automated process supervision alongside the weighted self-consistency algorithm, we have enhanced the instruction tuned Gemini Pro model's math reasoning performance, achieving a 69.4\% success rate on the MATH benchmark, a 36\% relative improvement from the 51\% base model performance. Additionally, the entire process operates without any human intervention, making our method both financially and computationally cost-effective compared to existing methods.
△ Less
Submitted 5 June, 2024;
originally announced June 2024.
-
Mapping Literacies in the Tourism Labor Market: A Cross-Database Comparison
Authors:
Eddy Soria Leyva,
Ana Beatriz Hernandez Lara
Abstract:
This book chapter conducts a comparative bibliometric analysis of literacies in the tourism labor market, drawing from the Web of Science (WoS) and Scopus databases. The objective is to assess scientific outputs and identify key patterns of scientific collaboration. Findings suggest a statistically significant difference between the two databases with an overlap level of 35.71%. However, there is…
▽ More
This book chapter conducts a comparative bibliometric analysis of literacies in the tourism labor market, drawing from the Web of Science (WoS) and Scopus databases. The objective is to assess scientific outputs and identify key patterns of scientific collaboration. Findings suggest a statistically significant difference between the two databases with an overlap level of 35.71%. However, there is a gradual and correlated increase in the number of publications over time. Scopus stands out for its broader impact and enduring citation relevance, suggesting its academic contributions have a longer-lasting effect. Conversely, WoS is characterized by a focus on more recent influential publications and exhibits a marginally more intense collaboration network.
△ Less
Submitted 23 February, 2024;
originally announced February 2024.
-
Leveraging Large Language Models in Conversational Recommender Systems
Authors:
Luke Friedman,
Sameer Ahuja,
David Allen,
Zhenning Tan,
Hakim Sidahmed,
Changbo Long,
Jun Xie,
Gabriel Schubiner,
Ajay Patel,
Harsh Lara,
Brian Chu,
Zexi Chen,
Manoj Tiwari
Abstract:
A Conversational Recommender System (CRS) offers increased transparency and control to users by enabling them to engage with the system through a real-time multi-turn dialogue. Recently, Large Language Models (LLMs) have exhibited an unprecedented ability to converse naturally and incorporate world knowledge and common-sense reasoning into language understanding, unlocking the potential of this pa…
▽ More
A Conversational Recommender System (CRS) offers increased transparency and control to users by enabling them to engage with the system through a real-time multi-turn dialogue. Recently, Large Language Models (LLMs) have exhibited an unprecedented ability to converse naturally and incorporate world knowledge and common-sense reasoning into language understanding, unlocking the potential of this paradigm. However, effectively leveraging LLMs within a CRS introduces new technical challenges, including properly understanding and controlling a complex conversation and retrieving from external sources of information. These issues are exacerbated by a large, evolving item corpus and a lack of conversational data for training. In this paper, we provide a roadmap for building an end-to-end large-scale CRS using LLMs. In particular, we propose new implementations for user preference understanding, flexible dialogue management and explainable recommendations as part of an integrated architecture powered by LLMs. For improved personalization, we describe how an LLM can consume interpretable natural language user profiles and use them to modulate session-level context. To overcome conversational data limitations in the absence of an existing production CRS, we propose techniques for building a controllable LLM-based user simulator to generate synthetic conversations. As a proof of concept we introduce RecLLM, a large-scale CRS for YouTube videos built on LaMDA, and demonstrate its fluency and diverse functionality through some illustrative example conversations.
△ Less
Submitted 16 May, 2023; v1 submitted 13 May, 2023;
originally announced May 2023.
-
Evaluation of Synthetic Datasets for Conversational Recommender Systems
Authors:
Harsh Lara,
Manoj Tiwari
Abstract:
For researchers leveraging Large-Language Models (LLMs) in the generation of training datasets, especially for conversational recommender systems - the absence of robust evaluation frameworks has been a long-standing problem. The efficiency brought about by LLMs in the data generation phase is impeded during the process of evaluation of the generated data, since it generally requires human-raters…
▽ More
For researchers leveraging Large-Language Models (LLMs) in the generation of training datasets, especially for conversational recommender systems - the absence of robust evaluation frameworks has been a long-standing problem. The efficiency brought about by LLMs in the data generation phase is impeded during the process of evaluation of the generated data, since it generally requires human-raters to ensure that the data generated is of high quality and has sufficient diversity. Since the quality of training data is critical for downstream applications, it is important to develop metrics that evaluate the quality holistically and identify biases. In this paper, we present a framework that takes a multi-faceted approach towards evaluating datasets produced by generative models and discuss the advantages and limitations of various evaluation methods.
△ Less
Submitted 12 December, 2022;
originally announced December 2022.
