Hypothesis Testing Prompting Improves Deductive Reasoning in Large Language Models

Abstract

Combining different forms of prompts with pre-trained large language models has yielded remarkable results on reasoning tasks (e.g. Chain-of-Thought prompting). However, along with testing on more complex reasoning, these methods also expose problems such as invalid reasoning and fictional reasoning paths. In this paper, we develop Hypothesis Testing Prompting, which adds conclusion assumptions, backward reasoning, and fact verification during intermediate reasoning steps. Hypothesis Testing prompting involves multiple assumptions and reverses validation of conclusions leading to its unique correct answer. Experiments on two challenging deductive reasoning datasets ProofWriter and RuleTaker show that hypothesis testing prompting not only significantly improves the effect, but also generates a more reasonable and standardized reasoning process.

Keywords: Deductive Reasoning, Large Language Models, Prompt

1.   Introduction

The release of large language models (LLMs) has revolutionized the NLP landscape recently Thoppilan et al. (2022); Kaplan et al. (2020); Chowdhery et al. (2022). Scaling up the size of language models and conducting diversified prompt methods become mainstream Liu et al. (2023c); Wei et al. (2022a); Yang et al. (2023). Given In-context learning or Chain-of-Thought prompts have already achieved high performance on challenging tasks such as commonsense, arithmetic, and symbolic reasoning Imani et al. (2023); Lee et al. (2021); Kojima et al. (2022). Logical reasoning is one of the most important and long-standing problems in NLP Hirschberg and Manning (2015); Russell and Norvig (2010), and integrating this ability into natural language understanding systems has always been a goal pursued Du et al. (2022).

Nevertheless, scaling has been demonstrated to offer limited advantages in resolving complex logical reasoning issues Kazemi et al. (2022). For example, Saparov and He (2022) show that Chain-of-Thought prompting struggles with proof planning for more complex logical reasoning problems. Additionally, the performance suffers greatly while handling recently released and out-of-distribution logical reasoning datasets Liu et al. (2023a). Despite many works have explored variants of Chain-of-Thought prompts to facilitate LLMs inference Zelikman et al. (2022); Zheng et al. (2023), we discover that the present logical reasoning task prompts place an excessive amount of emphasis on the reasoning process while ignoring the origin, purpose, and effectiveness of reasoning Creswell et al. (2022); Xi et al. (2023). As examples shown in Figure 1, the difficulty in judging logical problems arises not only from the process of reasoning but also from the choice of facts and rules to use as a starting point. Even if we were provided the thought process for some of the issues, it would not be very beneficial for others, based on how we previously created the prompts.

Figure 1: Questions in RuleTaker involve logical reasoning with facts and rules.

In this paper, we propose Hypothesis Testing Prompting, a new and more considerate prompt template design idea. Hypothesis testing is a formal procedure for investigating our ideas about the world using statistics and is often used by scientists to test specific predictions Bevans (2022). We draw inspiration from its process to introduce a process of conclusion assumptions, backward reasoning, and fact verification. Experiments on RuleTaker Clark et al. (2020) and ProofWriter Tafjord et al. (2021) show the effectiveness of our novel prompting paradigm as a strategy for promoting deductive reasoning in large language models. Further analyses show that Hypothesis Test prompting generates more desirable intermediate processes and significantly improves the "Unknown" label.

2.   Related Work

2.1.   Few-Shot Prompting

Brown et al. (2020) propose in-context learning as an alternative few-shot prompting way to stimulate ability. Besides, chain-of-Thought (CoT) Wei et al. (2022b) is one of the most well-known works, which decomposes the problem into intermediate steps and further improves the ability of large language models. Subsequently, several follow-up works were carried out, including Zero-shot-CoT (simply adding "Let’s think step by step" before each answer) Kojima et al. (2022), Self-consistency Wang et al. (2022), complexity-based Fu et al. (2022), and other prompting work Liu et al. (2023b); Jung et al. (2022); Zhou et al. (2022); Saparov and He (2022). While these methods enhance the performance of inference by paying attention to indications of the reasoning process, they often overlook some aspects such as identifying the root cause of the problem, establishing efficient reasoning strategies, and determining the direction of logical reasoning.

2.2.   Deductive Reasoning

Deductive reasoning is defined as the application of general concepts to particular circumstances Johnson-Laird (2010). Making logical assumptions is the foundation of deductive reasoning, which then bases a conclusion on those assumptions. The deduction task is then applied to a situation from the actual world after starting with a rule. In light of the principles "All men are mortal." and "Socrates is a man." for example, we can draw the conclusion that "Socrates is mortal." Johnson-Laird (1999).

3.   Hypothesis Testing Prompting

Hypothesis testing is a formal procedure for investigating our ideas about the world using statistics and used by scientists to test specific predictions that arise from theories Bevans (2022); La et al. (2012). There are 5 main steps in hypothesis testing:

1. State your research hypothesis;

2. Collect data in a way designed to test the hypothesis;

3. Perform an appropriate statistical test;

4. Decide whether to reject or fail to reject your null hypothesis;

5. Present the findings in your results and discussion section;

When completing a challenging reasoning activity, such as a multi-step deductive reasoning problem, one is not conducting random reasoning to obtain all possible intermediate results. We shall choose the relevant conditions for inference verification after initially making assumptions about the judgment problem, such as " First assume the conclusion is True and start from … Then assume the conclusion is False and start from … because the rules state that … So the conclusion …". The purpose of this study is to give language models the capacity to build a process that is similar to what we defined as Hypothesis Testing Prompting. We will show that large language models can generate more appropriate thought and more accurate results if demonstrations of hypothesis test prompting are provided in the exemplars for few-shot prompting. Figure 2 shows an example of a model producing a hypothesis testing thought to solve a deductive reasoning problem.

Figure 2: Comparison of three prompting methods: (a) Standard (b) Chain-of-Thought (c) Hypothesis Testing. Particularly, we highlight the Hypothesis testing reasoning processes. The comparative experimental results show that: Hypothesis testing prompting enables large language models to tackle complex logical reasoning.

4.   Experiment

4.1.   Experimental Setup

We explore Hypothesis Test Prompting for ChatGPT (GPT-3.5-Turbo in the OpenAI API) on multiple logical reasoning benchmarks.

Benchmarks. Considering FOL reasoning in question answering systems, there are two world assumptions Reiter (1981) that result in different objectives. One is the closed world assumption (CWA), which is the presumption that what is not currently known to be entailment is contradiction. The other is the open world assumption (OWA), whose objective should distinguish false propositions from uncertain ones. Due to differences in world assumptions, our analysis and solutions are also different.

We consider the following two deductive reasoning problem benchmarks: (1) the RuleTaker Clark et al. (2020) benchmark using CWA assumption; (2) the ProofWriter Tafjord et al. (2021) benchmark using OWA assumption. Both datasets are divided into five parts, each part requiring 0, $\leq$ 1, $\leq$ 2, $\leq$ 3, and $\leq$ 5 hops of reasoning, respectively. We conducted comparison tests on the test set of the two datasets for 5 distinct hops.

Standard prompting. As one of the baselines, we take into account the common few-shot prompting, made popular by Brown et al. (2020), in which a language model is provided with in-context examples of input-output pairings before producing a prediction for a test-time example. Examples are presented in the form of questions and answers. As seen in Figure 2(above), the model directly answers the question.

Chain-of-Thought prompting. We also compare with Chain-of-thought prompting which has achieved encouraging results on complex reasoning tasks Wei et al. (2022b). As seen in Figure 2(middle), the model not only provides the final answer but also comes with the consideration of intermediate steps.

Figure 3: Prediction accuracy results on the (a) RuleTaker and (b) ProofWriter datasets.

Hypothesis Testing Prompting. Our proposed approach is to augment each exemplars in few-shot prompting with the thought of hypothesis testing for an associated answer, as illustrated in Figure 2(below). We show one chain of thought exemplars (Example: Judge the following conclusion ’<Conclusion>’ is true, false, or unknown, based on the following facts and rules: <Facts> … <Rules> …).

4.2.   Experimental Results

The results for Hypothesis Testing Prompting and the baselines on the RuleTaker datasets are provided in Figure 3(a), and ProofWriter results are shown in Figure 3(b). From the results, we observe that our method significantly outperforms the other two baselines, especially on ProofWriter. Figure 3(a) demonstrates that while CoT performs well in the low hop, Hypothesis Testing prompting performs better as the hops count increases on RuleTaker. While on ProofWriter, our approach has a thorough lead (improved accuracy by over 4% on all hops). Comparing two datasets, the latter distinguishes between "False" and "Unknown", which demand a greater level of logic. The results on two datasets that were analyzed show a weakness in all methods for handling "Unknown" labels. This beacuse the OWA hypothesis necessitates the exclusion of both positive and negative findings to validate the "Unknown" label. The advantages of our strategy are illustrated by the comparison of the model output outputs in Figure 2. The content "First assume the conclusion is True … Then assume the conclusion is False … So … is Unknown." generated by the model through learning Hypothesis Testing prompting is more in line with our thinking. Besides, we’ll conduct further research and show it later.

(a) The proof accuracy of Chain-of-Thought and Hypothesis Testing prompting.

(b) Comparison of accuracy between Chain-of-Thought and Hypothesis Testing prompting on "Unknown" label.

Figure 4: Further results on ProofWriter.

4.3.   Further Analysis

We carry out the following thorough analysis to better comprehend the thought process:

Proof Accuracy. Five students are required to manually evaluate the outcomes of the intermediate reasoning after we randomly picked 100 examples from depth-5 of the ProofWriter. Proof accuracy represents the proportion where the inference process has been proven to be reasonable in the correct part of data label prediction. We compare the results of Chian-of-Thought and Hypothesis Testing prompting and report in Figure 4(a). While Hypothesis Testing prompting mostly produced the correct intermediate reasoning process when the predicted label was correct, CoT only generated the correct chain for 26% of the examples. This result is in line with other research showing that LMs rely on spurious correlations when solving logical problems from beginning to end. Additionally, our approach can successfully increase reasoning’s rationality. In processing the "Unknown" label, Hypothesis Testing prompting performs noticeably better than Chain-of-Thought.

"Unknown" accuracy. In the ProofWriter dataset, we separately counted the accuracy of the "Unknown" label shown in Figure 4(b). The results point to a flaw in the Chain-of-Thought strategy’s handling of "Unknown" labels(only 0.3 accuracyies). Contrarily, Hypothesis Testing prompting significantly increases the reliability of judging this label (up to 0.65). This further illustrates the value of holding various assumptions, as well as the reverse confirmation of conclusions.

5.   Conclusion

We have investigated Hypothesis Testing prompting as a straightforward and widely applicable technique for improving deductive reasoning in large language models. Multiple assumptions are made during hypothesis testing, and conclusions are reverse-validated to arrive at the one and only accurate answer. Through experiments on two logical reasoning datasets, we find that Hypothesis Testing prompting allows large language models to construct reasoning more reasonably and accurately. We anticipate that additional research on language-based reasoning approaches will be stimulated by our novel prompting design strategy.

6.   References