Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models
Step-Back Prompting Enables Reasoning Via Abstraction in Large Language Models
Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models
Abstract
We present Step-Back Prompting, a simple prompting technique that enables LLMs to do abstractions to derive high-level concepts and first principles from instances containing specific details. Using the concepts and principles to guide reasoning, LLMs significantly improve their abilities in following a correct reasoning path towards the solution. We conduct experiments of Step-Back Prompting with PaLM-2L, GPT-4 and Llama2-70B models, and observe substantial performance gains on various challenging reasoning-intensive tasks including STEM, Knowledge QA, and Multi-Hop Reasoning. For instance, Step-Back Prompting improves PaLM-2L performance on MMLU (Physics and Chemistry) by and respectively, TimeQA by , and MuSiQue by .
The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise. — Edsger W. Dijkstra
1 Introduction
The field of natural language processing (NLP) is witnessing a ground-breaking revolution because of the Transformer-based (Vaswani et al., 2017) large language models (LLMs) (Devlin et al., 2018; Raffel et al., 2020; Brown et al., 2020; Anil et al., 2023). Scaling up the model size and pre-training corpus (Hoffmann et al., 2022; Chowdhery et al., 2022) has brought remarkable improvement in model capabilities and sample efficiency with insights from the scaling law (Kaplan et al., 2020; Hoffmann et al., 2022), as well as emergent abilities (Wei et al., 2022a) such as multi-step reasoning (Wei et al., 2022b; Zhou et al., 2022) and instruction following (Mishra et al., 2022b; Wei et al., 2021).
Despite the great advancements, complex multi-step reasoning remains challenging for even the state-of-the-art LLMs. Lightman et al. (2023) show that process-supervision with step-by-step verification is a promising remedy to improve the correctness of intermediate reasoning steps. Techniques such as Chain-of-Thought (Wei et al., 2022b) were introduced to produce a coherent series of intermediate reasoning steps to increase the success rate of following the right decoding path. Inspired by the fact that when faced with challenging tasks humans often step back and do abstractions to arrive at high-level principles to guide the process, we propose Step-Back Prompting to ground reasoning on abstractions to reduce the chance of making errors in the intermediate reasoning steps.
Among many of the cognitive skills, abstraction (Lachmy et al., 2022) is ubiquitous to humans’ ability to process vast amounts of information and derive general principles. For example, Kepler compressed thousands of measurements into Kepler’s three laws of planetary motion, which precisely describe the orbits of planets around the Sun (Russell, 1964). In critical decision-making, humans find abstraction to be helpful since it provides a broader view of the environment. This work explores how LLMs can tackle complex tasks involving many low-level details through a two-step process of abstraction-and-reasoning. The first step is to show LLMs how to step back through in-context learning – prompting them to derive high-level abstractions such as concepts and principles for a specific example. The second step is to leverage the reasoning ability to reason on top of the high-level concepts and principles. We use few-shot exemplar demonstrations to execute Step-Back Prompting on LLMs.
We experiment across a range of tasks involving domain specific reasoning such as Physics and Chemistry, knowledge-intensive question answering requiring factual knowledge, multi-hop commonsense reasoning. We observe significant performance improvements (up to ) in PaLM-2L (Anil et al., 2023) demonstrating the efficacy of Step-Back Prompting in tackling complex tasks, which are otherwise challenging due to the amount of details needed for reasoning. Figure 1 shows a summary of all the key results presented in this paper. Some the tasks are very challenging: both PaLM-2L and GPT-4 achieve only accuracy on TimeQA and MuSiQue. Chain-of-Thought prompting leads to a minor improvement on a few tasks, while Step-Back Prompting improves the performance of PaLM-2L across the board: and on MMLU Physics and Chemistry, on TimeQA, and on MuSiQue.
We conduct a variety of analyses and find that Step-Back Prompting leads to strong performance improvements (up to ) over chain-of-thought (CoT) prompting (Wei et al., 2022b) and “take-a-deep-breath” (TDB) prompting (Yang et al., 2023). We perform a qualitative evaluation where we find that Step-Back fixes a large portion of errors of the base model (up to ) while introducing a small portion of new errors (max ). We also conduct an error analysis and find that majority of the errors made by Step-Back Prompting is attributed to the intrinsic limitations of reasoning capabilities of LLMs while abstraction skills are relatively easy to demonstrate to LLMs, pointing out the direction for future improvements of methods alike Step-Back Prompting.
2 Step-Back Prompting
Step-Back Prompting is motivated by the observation that many tasks contain a lot of details, and it is hard for LLMs to retrieve relevant facts to tackle the task. As shown in the first example (top) in Figure 2, for a Physics question of “What happens to the pressure, P, of an ideal gas if the temperature is increased by a factor of 2 and the volume is increased by a factor of 8 ?”, the LLM can deviate from the first principle of Ideal Gas Law when reasoning directly on the question. Similarly, a question of “Estella Leopold went to which school between Aug 1954 and Nov 1954?” is very hard to address directly given the detailed time range constraint. In both cases, asking a step-back question helps the model to solve the problem effectively.
We define a step-back question as a derived question from the original question at a higher level of abstraction. For instance, instead of directly asking “which school Estella Leopold went to during a specific period”, a step-back question (Figure 2 bottom) would ask about the “education history”, which is a high-level concept encompasses the original question. Answering the step-back question of “Estella Leopold’s education history” in this case will provide all the necessary information to reason about “which school Estella Leopold went to during a specific period”. The premise is that the step-back question is typically much easier. Grounding the reasoning on top of such abstractions helps to avoid reasoning errors in the intermediate steps such as the example shown in Figure 2 (left) from Chain-of-Thought. In short, Step-Back Prompting consists two simple steps:
-
•
Abstraction: Instead of addressing the question directly, we first prompt the LLM to ask a generic step-back question about a higher-level concept or principle, and retrieve relevant facts about the high-level concept or principle. The step-back question is unique for each task in order to retrieve the most relevant facts.
-
•
Reasoning: Grounded on the facts regarding the high-level concept or principle, the LLM can reason about the solution to the original question. We term this as Abstraction-grounded Reasoning.
In the following sections, we present an empirical study of Step-Back Prompting on a range of challenging tasks covering STEM, Knowledge QA, and Multi-Hop Reasoning involving complex reasoning.
3 Experimental Setup
Here we define the tasks and models we experiment with. We also describe our evaluation metric and the baselines we consider.
3.1 Tasks
We experiment with the following diverse tasks: (a) STEM, (b) Knowledge QA, and (c) Multi-Hop Reasoning. We describe below the datasets we consider (see Appendix B for more details).
-
•
STEM: We evaluate MMLU and GSM8K for STEM tasks. MMLU (Hendrycks et al., 2020) contains a series of benchmarks across diverse domains to evaluate the model’s language understanding. We consider the high school physics and chemistry portions of MMLU because of the deep reasoning involved.
-
•
Knowledge QA: We consider TimeQA (Chen et al., 2021) since it contains complex queries that require challenging time-sensitive knowledge. We also experiment with SituatedQA (Zhang & Choi, 2021), another challenging open-retrieval QA dataset requiring the model to answer questions given temporal or geographical contexts.
- •
3.2 Models
3.3 Evaluation
Conventional evaluation metrics such as accuracy, F1 score have limitations specifically for evaluating the generations of state-of-the-art LLMs since these models often generate long-form answers which are hard to capture. We instead conduct an evaluation using the PaLM-2L model where we few-shot prompt the model to identify equivalence between target answers and the model predictions. Few-shot examples, prompts and other details used for this evaluation are in Appendix C.
3.4 Baseline Methods
-
•
PaLM-2L, PaLM-2L 1-shot: PaLM-2L is either queried directly with the question or has a single demonstration exemplar of question-answer included in the prompt.
-
•
PaLM-2L + CoT, PaLM-2L + CoT 1-shot: PaLM-2L model is queried with zero-shot CoT prompting (Kojima et al., 2022): “Let’s think step by step” is appended to the question. For 1-shot, One demonstration example of a question and answer pair is provided in the prompt, where the answer is in the style of CoT (Wei et al., 2022b).
-
•
PaLM-2L + TDB: Zero-shot prompting with “Take a deep breath and work on this problem step-by-step.” (Yang et al., 2023) prepended to the question.
- •
-
•
GPT-4 and Llama2-70B: we run GPT-4 and Llama2-70B on MMLU tasks for all methods. In addition, we also run GPT-4 on all baselines for all tasks.
We do not use RAG for STEM tasks, because of the inherent reasoning nature of the tasks contrary to the other fact-seeking datasets. All inferences are done using greedy decoding.
Method | MMLU Physics | MMLU Chemistry |
---|---|---|
PaLM-2L | 66.4% (0.8%) | 70.9% (0.9%) |
PaLM-2L 1-shot | 64% (1.6%) | 75.6% (0.4%) |
PaLM-2L + CoT | 65% (2%) | 75.3% (1.5%) |
PaLM-2L + CoT 1-shot | 61.5% (1.8%) | 76.6% (1%) |
PaLM-2L + TDB | 65.7% (0.7%) | 73.8% (1.1%) |
PaLM-2L + Step-Back (ours) | 73.2% (1.9%) | 81.8% (1.4%) |
GPT-4 | 69.4% (2.0%) | 80.9% (0.7%) |
GPT-4 1-shot | 78.4% (2.4%) | 80.5% (1.6%) |
GPT-4 + CoT | 82.9% (0.5%) | 85.3% (1.0%) |
GPT-4 + CoT 1-shot | 79.3% (1.0%) | 82.8% (0.5%) |
GPT-4 + TDB | 74.4% (4.0%) | 81.5% (1.3%) |
GPT-4 + Step-Back (ours) | 84.5% (1.2%) | 85.6% (1.4%) |
Llama2-70B | 51.9% (3.6%) | 55.7% (2.1%) |
Llama2-70B 1-shot | 57.3% (1.6%) | 58.5% (2.5%) |
Llama2-70B + CoT | 59.3% (2.0%) | 64.1% (1.2%) |
Llama2-70B + CoT 1-shot | 59.6% (2.0%) | 68.1% (1.4%) |
Llama2-70B + TDB | 60.4% (2.1%) | 63.6% (1.9%) |
Llama2-70B + Step-Back (ours) | 64.8% (1.5%) | 66.7% (1.6%) |
4 STEM
We evaluate Step-Back Prompting on STEM tasks (Hendrycks et al., 2020) to gauge the efficacy of our method on reasoning in highly specialized domains. We explain below our experimental setup, result, and analysis of applying Step-Back Prompting on the MMLU high-school Physics and Chemistry, and GSM8K benchmarks.
4.1 Step-Back Prompting
Questions in the MMLU benchmarks require deeper reasoning. Furthermore, they also require understanding and application of formulae which are often physics and chemistry principles and concepts. In this case, we first demonstrate to the model abstraction skills in the form of concepts and first principles such as Newton’s first law of motion, Doppler effect, and Gibbs free energy etc. The implicit step-back question here is “what are the physics or chemistry principles and concepts involved in solving this task?”. We provide demonstrations to the model to recite the relevant principles for solving the task from its own knowledge (see Appendix D.1 for few-shot exemplars).
4.2 Results
Table 1 illustrates model performance across various setups across three model families: PaLM-2L, GPT-4, and Llama2-70B. Average accuracy over 5 evaluation runs is reported along with standard deviations (in the parentheses). PaLM-2L baseline performance is and 70.9% on Physics and Chemistry, respectively. We find that CoT and TDB zero-shot prompting do not significantly increase model performance, which could be due to the inherent difficulty and deep reasoning associated with these tasks. PaLM-2L 1-shot and PaLM-2L + CoT 1-shot do not improve against the baseline much either, highlighting the challenge of demonstrating the reasoning steps to the model. In contrast, Step-Back Prompting significantly improves model performance: +7% and +11% compared to PaLM-2L. Similarly, with GPT-4 and Llama2-70B models, Step-Back Prompting is very competitive among all the baseline methods we tested, showing that Step-Back Prompting is model-agnostic. We present the results of GSM8K in Appendix A.1.
4.3 Ablation and Analysis
Few-shot Ablation: First, in Figure 3, we observe that Step-Back Prompting is robust to the number of few-shot exemplars of (question, principles) pairs used as demonstrations. Adding more demonstration examples beyond a single example does not lead to further improvements. This indicates that the task of retrieving the relevant principles and concepts is relatively easy through in-context learning and a single demonstration suffices. Therefore, we use a single exemplar for few-shot prompting throughout the paper except the ablation studies.
Error Analysis: Comparing the predictions of Step-Back Prompting to the baseline PaLM-2L model for MMLU high-school Physics: we find that Step-Back Prompting corrects errors from the baseline while introducing errors.
To further understand where the errors come from in Step-Back Prompting, we annotate all the wrong predictions of Step-Back Prompting in the test set, and categorize them into 5 classes (see Appendix E.1 for examples in each class):
-
•
Principle Error: The error happens at the step of Abstraction, where the first principles generated by models are wrong or incomplete.
-
•
Factual Error: There is at least one factual error when the model recites its own factual knowledge
-
•
Math Error: There is at least one math error in the intermediate steps when math calculations are involved in deriving the final answer.
-
•
Context Loss: There is at least one error where the model response loses context from the question, and deviates from addressing the original question
-
•
Reasoning Error: We define Reasoning Error as when the model makes at least one error in the intermediate Reasoning steps before arriving at the final answer.
All five types of errors are happening during the Reasoning step except Principle Error which points to the failure of the Abstraction step. As shown in Figure 4 (right), Principle Error comprises only a small fraction of the errors the model makes: more than of the errors happen at the Reasoning step. Among the four error types during Reasoning, Reasoning Error and Math Error are the major error categories. This corroborates with the finding in the ablation study above that very few exemplars are needed to demonstrate to LLMs the Abstraction skill. Reasoning step is still the bottleneck of how well Step-Back Prompting can perform tasks such as MMLU requiring complex reasoning. For MMLU Physics specifically, the Reasoning and Math skills are critical for solving the problems successfully: even if the first principles are retrieved correctly, deep reasoning and math are involved to derive a correct final answer through a typical multi-step reasoning process.
Method | TimeQA | TQA Easy | TQA Hard | SituatedQA |
---|---|---|---|---|
PaLM-2L | 41.5% | 42.6% | 40.4% | 54.3% (0.3%) |
PaLM-2L 1-shot | 40.7% | 41.7% | 39.1% | 51.8% (0.6%) |
PaLM-2L + CoT | 40.8% | 41.8% | 39.8% | 56.4% (0.2%) |
PaLM-2L + CoT 1-shot | 38.1% | 39.3% | 36.8% | 54% (0.8%) |
PaLM-2L + TDB | 40.9% | 42.6% | 39.1% | 54% (0.5%) |
PaLM-2L + RAG | 57.4% | 67.8% | 46.8% | 59.3% (0.4%) |
PaLM-2L + Step-Back (ours) | 66% | 70.4% | 61.6% | 57.5% (0.3%) |
PaLM-2L + Step-Back + RAG (ours) | 68.7% | 75.2% | 62.3% | 61% (0.4%) |
GPT-4 | 45.6% | 48.9% | 42.6% | 63.2% (0.4%) |
5 Knowledge QA
We evaluate Step-Back Prompting on question-answering benchmarks requiring intensive factual knowledge. Knowledge QA has been challenging for LLMs. In this section, we first describe the experimental setup, followed by results and analysis on Step-Back Prompting.
5.1 Step-Back Prompting
We evaluate Step-Back Prompting on TimeQA (Chen et al., 2021) and SituatedQA (Zhang & Choi, 2021) in the Knowledge QA category. We first show the LLMs how to do Abstraction through in-context demonstrations. The step-back question “What was Estella Leopold’s education history” in Figure 2 is generated by the LLM through few-shot demonstrations (see Appendix D.2 for details). Given the knowledge-intensive nature of these queries, we use retrieval augmentation (RAG) in combination with Step-Back Prompting. The step-back question is used to retrieve relevant facts, which work as additional context (see Table 14 for the prompt) to ground the final reasoning step.
5.2 Results
We evaluate the models on the test set of TimeQA. As shown in Table 2, the baseline models of GPT-4 and PaLM-2L achieved and , highlighting the difficulty of the task. Applying either CoT or TDB zero-shot (and one-shot) prompting to the baseline model shows no improvement. In contrast, augmenting the baseline model by regular retrieval augmentation (RAG) improves the accuracy to , highlighting the fact-intensive nature of the task. The result of Step-Back + RAG shows the effectiveness of going back to a high-level concept, which enables much more reliable retrieval augmentation: the accuracy on TimeQA achieves a remarkable .
Next, we segment TimeQA into the Easy and Hard difficulty levels provided in the original dataset. As expected, all methods perform worse on the Hard subset. While RAG can improve the Easy accuracy from to , the improvement is much smaller on the Hard accuracy: to . This is where Step-Back Prompting shines by retrieving facts regarding high-level concepts to ground the final reasoning: Step-Back + RAG further improves the Hard accuracy to , outperforming GPT-4’s from GPT-4. We hypothesize that facts regarding the high-level concepts (such as education history) are much more accessible than the low-level details.
On the SituatedQA benchmark, we observe a moderate quality gain from to our best method of Step-Back + RAG () with a small gap to GPT-4’s . Similar to TimeQA, prompting techniques such as CoT and TDB don’t help significantly for SituatedQA.
5.3 Ablation and Analysis
Few-shot Ablation: We observe in Figure 5 (left) that the performance of Step-Back Prompting on TimeQA is robust to the number of exemplars used in demonstration, highlighting again the sample efficiency of in-context learning Abstraction skills for models like PaLM-2L.
Error Analysis: Figure 5 (right) shows the breakdown of all the remaining errors made by Step-Back Prompting on TimeQA. Similar to Section 4.3, we categorize the errors into
-
•
StepBack: The step-back question generated is not helpful in solving the task.
-
•
RAG: RAG fails to retrieve relevant information despite that the step-back question is on target.
-
•
Scoring Error: The evaluation by the judge model made a mistake.
-
•
Reasoning Error: The retrieved context is relevant, but the model still fails to reason through the context to arrive at the right answer.
We find that the StepBack rarely fails. In contrast, we find more than half of the errors are due to reasoning errors. Additionally, of errors are due to failure in retrieving the right information despite that Abstraction provided by step-back makes it a much easier task. This reflects the difficulty level of the TimeQA task. Additional error analysis of TimeQA is in Appendix A.
Method | MuSiQue | StrategyQA |
---|---|---|
PaLM-2L | 35.5% (3%) | 82.8% (0.7%) |
PaLM-2L 1-shot | 29.0% (0.5%) | 76.6% (0.5%) |
PaLM-2L + CoT | 38.7% (3.2%) | 83.6% (0.4%) |
PaLM-2L + CoT 1-shot | 38.5% (2.2%) | 76.8% (1.4%) |
PaLM-2L + TDB | 39.0% (2.3%) | 82.7% (0.9%) |
PaLM-2L + RAG | 39.6% (2.8%) | 84.2% (0.5%) |
PaLM-2L + Step-Back (ours) | 42.6% (3.1%) | 82.7% (0.4%) |
PaLM-2L + Step-Back + RAG (ours) | 42.8% (2.0%) | 86.4% (1%) |
GPT-4 | 38.5% (0.2%) | 78.3% (1.1%) |
6 Multi-Hop Reasoning
We evaluate Step-Back Prompting on challenging Multi-Hop reasoning benchmark MuSiQue (Trivedi et al., 2022) and StrategyQA (Geva et al., 2021). We follow the same protocol as Section 5 to implement Step-Back Prompting.
Table 3 shows performance of various baselines on the dev set of MuSiQue and StrategyQA. Baseline performance of PaLM-2L and GPT-4 are low ( and for PaLM-2L and GPT-4 respectively) in MuSiQue since it is a hard multihop reasoning benchmark. In contrast, StrategyQA has stronger baselines ( and for PaLM-2L and GPT-4 respectively) probably because it is a binary classification task. CoT and TDB improve model performance a bit in the case of MuSiQue ( 3% and 3.5% respectively) which can be attributed to the inherent reasoning nature of this task where these methods are shown to be helpful. In the case of StrategyQA, there is no significant performance gain with CoT and TDB which could be due to the high baseline performance in this task, with limited scope for these prompting methods to improve performance. Often, 1-shot performance is significantly lower than their zero-shot methods, which could be attributed to potential example bias (Zhao et al., 2021; Parmar et al., 2023). RAG improves model performance ( 4% and 2% for MuSiQue and StrategyQA respectively.) Step-Back Prompting with the power of abstraction produces the best performance of all methods: in MuSiQue and in StrategyQA, significantly outperforming GPT-4 on both tasks. We present a detailed error analysis on StrategyQA in Appendix A.3.
7 Discussion
Abstraction helps humans to solve complex tasks by removing irrelevant details and distilling high-level concepts and principles to guide the problem-solving process. Step-Back Prompting breaks complex tasks such as knowledge-intensive QA, multi-hop reasoning, and science questions into two separate steps of Abstraction and Reasoning. We demonstrate through empirical experiments that Abstraction is an easy skill for the LLMs such as PaLM-2L via sample-efficient in-context learning. Grounding on the high-level concepts and principles, LLMs can leverage their intrinsic Reasoning capabilities to derive the solution. This reduces the chance of reasoning failures in the intermediate steps and is shown to improve the performance on a wide range of complex reasoning tasks. Despite the success, through error analysis, we find that Reasoning is still one of the hardest skills for LLMs to acquire: it is still the dominant failure mode even after the large reduction of task complexity by Step-Back Prompting.
Nevertheless, Abstraction is neither necessary nor possible in all scenarios. For instance, the task can be as simple as who was the president of the United States in 2000?, in which case there is no such need to step back and ask a high-level question as the answer to such questions is readily available. Questions such as what is the speed of light? point to the first principles themselves. Doing Abstraction in this case would not make a difference either.
8 Related Work
8.1 Prompting
Few-shot prompting (Brown et al., 2020; Liu et al., 2023; Mishra et al., 2022a; Wei et al., 2022b) has significantly improved model performance across a range of tasks without requiring updating any model parameters. Our work Step-Back Prompting is in the same category as the chain-of-thought prompting (Wei et al., 2022b) and scratchpad (Nye et al., 2021) owing to its simplicity and generic nature. But our approach is focused on the key idea of abstraction which is inspired from the fact that taking a step back often helps humans in performing complex tasks. Our work is also related to the recitation-augmented language models (Sun et al., 2022); however in contrast to their work, we explicitly perform step-back and abstraction, with optional use of retrieval augmentation depending on the nature of the task at hand.
8.2 Decomposition
Decomposing a task into simpler tasks and solving these tasks to complete the original task has been an effective way (Zhou et al., 2022; Patel et al., 2022; Khot et al., 2022; Press et al., 2022) to improve model performance on complex tasks. Several prompting methods have been successful in this regard. Our work Step-Back Prompting, in contrast, is on making the question more abstract and high-level, which is different from decomposition that is often a low-level breakdowns of the original question. For instance, a generic question for which employer did Steve Jobs work for in 1990? could be what is the employment history of Steve Jobs? While decomposition would lead to sub-questions such as What was Steve Jobs doing in 1990?, Was Steve Jobs employed in 1990? and If Steve Jobs was employed, who was his employer? Furthermore, abstract questions such as what is the employment history of Steve Jobs? are often generic in nature to have a many-to-one mapping since many questions (e.g. which employer did Steve Jobs work for in 1990? and which employer did Steve Jobs work for in 2000?) can have the same abstract question. This is in contrast to decomposition where there is often a one-to-many mapping since there are multiple decomposed sub-problems necessary to solve a given question.
9 Conclusion
We introduce Step-Back Prompting as a simple yet generic method to elicit deep reasoning via abstraction in large language models. Experimentation on LLMs across fact-seeking, commonsense reasoning and domain-specific reasoning benchmarks shows that Step-Back Prompting significantly improves model performance. We hypothesize that abstraction helps models to hallucinate less and reason better, probably reflecting the true nature of the model which are often hidden while responding to the original question without abstraction. We hope our work will inspire more human-inspired approaches to elicit the hidden potential of large language models.
Acknowledgement
We thank Andrew Dai, Adams Yu and Hannah Rashkin of Google DeepMind for their insightful feedback on this paper.
References
- Anil et al. (2023) Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023.
- Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
- Chen et al. (2021) Wenhu Chen, Xinyi Wang, and William Yang Wang. A dataset for answering time-sensitive questions. arXiv preprint arXiv:2108.06314, 2021.
- Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
- Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
- Geva et al. (2021) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346–361, 2021.
- Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
- Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.
- Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
- Khot et al. (2022) Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406, 2022.
- Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213, 2022.
- Lachmy et al. (2022) Royi Lachmy, Valentina Pyatkin, Avshalom Manevich, and Reut Tsarfaty. Draw me a flower: Processing and grounding abstraction in natural language. Transactions of the Association for Computational Linguistics, 2022.
- Lightman et al. (2023) Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023.
- Liu et al. (2023) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35, 2023.
- Mishra et al. (2022a) Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. Reframing instructional prompts to gptk’s language. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 589–612, 2022a.
- Mishra et al. (2022b) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3470–3487, 2022b.
- Nye et al. (2021) Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021.
- OpenAI (2023) OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
- Parmar et al. (2023) Mihir Parmar, Swaroop Mishra, Mor Geva, and Chitta Baral. Don’t blame the annotator: Bias already starts in the annotation instructions. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 1771–1781, 2023.
- Patel et al. (2022) Pruthvi Patel, Swaroop Mishra, Mihir Parmar, and Chitta Baral. Is a question decomposition unit all we need? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 4553–4569, 2022.
- Press et al. (2022) Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022.
- Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
- Russell (1964) John L Russell. Kepler’s laws of planetary motion: 1609–1666. The British journal for the history of science, 2(1):1–24, 1964.
- Sun et al. (2022) Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. Recitation-augmented language models. arXiv preprint arXiv:2210.01296, 2022.
- Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
- Trivedi et al. (2022) Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554, 2022.
- Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
- Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
- Wei et al. (2022a) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022a.
- Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022b.
- Yang et al. (2023) Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint arXiv:2309.03409, 2023.
- Zhang & Choi (2021) Michael Zhang and Eunsol Choi. Situatedqa: Incorporating extra-linguistic contexts into qa. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7371–7387, 2021.
- Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pp. 12697–12706. PMLR, 2021.
- Zhou et al. (2022) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022.
Appendix A GSM8K Results, and Error Analysis
A.1 GSM8K Results
We present in Table 4 the results of Step-Back Prompting on GSM8K along with other strong baselines from PaLM-2L runs. We observe that Step-Back Prompting achieved competitive performance together with zero-shot CoT and 1-shot standard prompting. We hypothesize that the simplicity of principles (e.g. addition, subtraction, etc.) in GSM8K makes it not absolutely necessary to retrieve the principles first before reasoning. Nonetheless, we still find that Step-Back Prompting is the most competitive among all the prompting methods we tested, including the “Take a Deep Breath” prompting optimized for GSM8K in Yang et al. (2023) and Decomposed Prompting in Khot et al. (2022).
Method | GSM8K |
---|---|
PaLM-2L | 75.8% (0.2%) |
PaLM-2L 1-shot | 84.5% (0.4%) |
PaLM-2L + CoT | 84.4% (0.2%) |
PaLM-2L + CoT 1-shot | 81% (0.2%) |
PaLM-2L + TDB | 82.2% (0.2%) |
PaLM-2L + DP | 82.2% (0.08%) |
PaLM-2L + Step-Back (ours) | 84.3% (0.2%) |
A.2 TimeQA Error Analysis
We conduct error analysis to understand where Step-Back Prompting fixes the errors the baseline models make. Figure 6 shows that compared to the predictions of baseline PaLM-2L, Step-Back Prompting can fix of the predictions where the baseline prediction is wrong, while causing errors.Furthermore, Step-Back + RAG fixes errors coming from RAG. The of errors introduced by Step-Back Prompting to RAG is still relatively low (). Together, this shows that the Step-Back Prompting is helpful most of the time, signifying the need and effectiveness of doing Abstraction before directly addressing the original question.
A.3 StrategyQA Error Analysis
Figure 7 shows the error analysis of StrategyQA on the predictions of Step-Back + RAG against the baseline model and the raw retrieval augmentation variant of PaLM-2L. Compared to the baseline, Step-Back + RAG can turn wrong predictions into correct predictions, while leading to errors the other way around. Furthermore, Step-Back + RAG fixes errors coming from RAG. The errors introduced to RAG by Step-Back are just .
Appendix B Dataset Details
Table 5 shows the split and number of examples used for evaluations in TimeQA, StrategyQA, MMLU, and GSM8K.
Domain | Dataset | Split | Number of Examples |
---|---|---|---|
STEM | MMLU high-school Physics | Test | 151 |
MMLU high-school Chemistry | Test | 203 | |
GSM8K | Test | 1319 | |
Knowledge QA | TimeQA | Test | 5226 |
TimeQA Easy | Test | 2613 | |
TimeQA Hard | Test | 2613 | |
SituatedQA | Test | 2901 | |
Multi-hop Reasoning | MuSiQue | Dev | 2417 |
StrategyQA | Dev | 229 |
Appendix C Evaluation Details
C.1 Few-shot Examples for Evaluation with PaLM-2L
Given the model free-form outputs and the target label, we use one positive and one negative output as few-shot examples to demonstrate to the scoring model how to score the output. Table 6 illustrates the prompt we used for the scoring model. We parse out the “Yes” or “No” answer from the scoring model output as a TRUE or FALSE score of the model output.
Are the following two answers to the given question equivalent? Do not consider whether the answers are right or wrong, but only whether they are equivalent. Directly state ”Yes” or ”No”.
|
C.2 Hyper-parameters for Evaluation with PaLM-2L
We use PaLM-2L as the scoring model for evaluation. We experiment with different sampling temperatures, and find that gives us a highly-accurate evaluation. For example, we sampled test examples and the model predictions, and manually rated the correctness of the model scoring. We found that out of 4 trials, the model scoring agrees with human ratings , , and of the time.
Appendix D Prompts and Few shot Examples
D.1 STEM
For MMLU high-school Physics and Chemistry, we first prompt the model to generate the first principles behind the question. Using the generated first principles, we further prompt the model to generate the final answer through few-shot demonstrations The prompt generating first principles is shown in Table 7 for MMLU high-school Physics and Chemistry.
MMLU Physics/Chemistry First-Principle Prompt |
You are an expert at Physics/Chemistry. You are given a Physics/Chemistry problem. Your task is to extract the Physics/Chemistry concepts and principles involved in solving the problem. Here are a few examples:
|
After extracting the first principles of solving a particular question, we formulate the prompt in Table 8 to query the model for the final answer.
MMLU Physics/Chemistry Final Answer Prompt |
You are an expert at Physics/Chemistry. You are given a Physics/Chemistry problem and a set of principles involved in solving the problem. Solve the problem step by step by following the principles. Here are a few examples:
|
Tables 9-10 show one demonstration exemplar of Question-Principles-Answer triplets for MMLU high-school Physics and Chemistry, respectively. For GSM8K, given the simplicity of the principles, we directly combine the principles and the solution in the demonstration exemplar. Table 11 shows the exemplar we used in the paper.
Question |
A spherical conductor carries a net charge. How is this charge distributed on the sphere?
|
---|---|
Principles |
Coulomb’s Law: the force between two charged particles is proportional to the product of their charges and inversely proportional to the square of the distance between them, F = k * q1 * q2 / r, where F is the electric force, k is a constant, q1 and q2 are the charges the particles carry, and r is the distance between them. |
Answer |
Using the Principles of Coulomb’s Law, we can solve the problem as following:
Step 2: Apply Coulomb’s Law to find out what happens if there are other charges present.
Therefore, the correct answer is 1) The charge resides on the surface only; the distribution of charge on the surface depends on what other charged objects are near the sphere. |
Question |
A sample of an unknown chloride compound was dissolved in water, and then titrated with excess Pb(NO3)2 to create a precipitate. After drying, it is determined there are 0.0050 mol of precipitate present. What mass of chloride is present in the original sample?
|
---|---|
Principles |
Precipitation reactions: Precipitation reactions occur when two soluble salts are mixed and form an insoluble product, called a precipitate. The precipitate can be separated from the solution by filtration or centrifugation.
|
Answer |
Assuming the unknown chloride compound is MCl, where M represents the metal cation, the balanced chemical equation for the precipitation reaction is:
Since is in excess, MCl is the limiting reactant. The stoichiometry of the reaction indicates that 2 moles of MCl produce 1 mole of precipitate. Therefore, 0.0050 mol of corresponds to 0.010 mol of MCl.
The mass of chloride in the original sample can be calculated using the molar mass of chloride (35.45 g/mol):
The correct answer is 1) 0.355 g. |
Question |
Bella bought stamps at the post office. Some of the stamps had a snowflake design, some had a truck design, and some had a rose design. Bella bought 11 snowflake stamps. She bought 9 more truck stamps than snowflake stamps, and 13 fewer rose stamps than truck stamps. How many stamps did Bella buy in all? |
---|---|
Answer |
Let us find and apply the math principles to solve the problem step by step:
Step 1. Addition: Calculate the number of truck stamps.
Step 2. Subtraction: Calculate the number of rose stamps.
Step 3. Addition: Calculate the total number of stamps in all three colors.
Conclusion: Bella bought 38 stamps in all. |
D.2 Knowledge QA
We use the following prompting in Table 12 to demonstrate to the LLM on asking a step-back question for TimeQA and SituatedQA including up to 5 exemplar demonstrations of pairs of Original Question and Step-back Question.
Knowledge QA Step-Back Prompt |
You are an expert at world knowledge. Your task is to step back and paraphrase a question to a more generic step-back question, which is easier to answer. Here are a few examples:
|
Table 13 shows 5 exemplars from the Train split of TimeQA and SituatedQA as demonstrations of asking step-back questions.
dataset |
Original Question |
Step-back Question |
---|---|---|
TimeQA |
Which position did Knox Cunningham hold from May 1955 to Apr 1956? |
Which positions have Knox Cunningham held in his career? |
TimeQA |
Who was the spouse of Anna Karina from 1968 to 1974? |
Who were the spouses of Anna Karina? |
TimeQA |
Which team did Thierry Audel play for from 2007 to 2008? |
Which teams did Thierry Audel play for in his career? |
TimeQA |
What was the operator of GCR Class 11E from 1913 to Dec 1922? |
What were the operators of GCR Class 11E in history? |
TimeQA |
Which country did Sokolovsko belong to from 1392 to 1525? |
Which countries did Sokolovsko belong to in history? |
SituatedQA |
when was the last time a team from canada won the stanley cup as of 2002 |
which years did a team from canada won the stanley cup as of 2002 |
SituatedQA |
when did england last get to the semi final in a world cup as of 2019 |
which years did england get to the semi final in a world cup as of 2019? |
SituatedQA |
what is the biggest hotel in las vegas nv as of November 28, 1993 |
what is the size of the hotels in las vegas nv as of November 28, 1993 |
SituatedQA |
who has scored most runs in t20 matches as of 2017 |
What are the runs of players in t20 matches as of 2017 |
SituatedQA |
who is the highest paid player in the nba this season as of 2017 |
what is the salary of the high paid players in the nba this season as of 2017 |
The step-back question is extracted from the model output using the prompt. Using the step-back question, we do retrieval augmentation. Using both the retrieval augmentations from the original question and the step-back question, we formulate the final prompt to query the model for the final answer, as shown in Table 14.
Knowledge QA Final-Answer Prompt |
You are an expert of world knowledge. I am going to ask you a question. Your response should be comprehensive and not contradicted with the following context if they are relevant. Otherwise, ignore them if they are not relevant.
|
D.3 Multi-Hop Reasoning
For Multi-Hop Reasoning, we use the same prompting template as in Knowledge QA to ask the step-back question, and query for the final answer given the retrieval augmentations. Table 15 shows 5 demonstration exemplars for asking step-back questions from the Train split of MuSiQue and StrategyQA.
dataset |
Original Question |
Step-back Question |
---|---|---|
MuSiQue |
at year saw the creation of the region where the county of Hertfordshire is located? |
which region is the county of Hertfordshire located? |
MuSiQue |
Jan Šindel’s was born in what country? |
what is Jan Šindel’s personal history? |
MuSiQue |
When was the abolishment of the studio that distributed The Game? |
which studio distributed The Game? |
MuSiQue |
What city is the person who broadened the doctrine of philosophy of language from? |
who broadened the doctrine of philosophy of language |
MuSiQue |
When was the baseball team winning the world series in 2015 baseball created? |
which baseball team won the world series in 2015 baseball? |
StrategyQA |
Could the members of The Police perform lawful arrests? |
what can the members of The Police do? |
StrategyQA |
Would a Monoamine Oxidase candy bar cheer up a depressed friend? |
What are the effects of Monoamine Oxidase? |
StrategyQA |
Would a dog respond to bell before Grey seal? |
Would a dog respond to bell before Grey seal? |
StrategyQA |
Is shrimp scampi definitely free of plastic? |
what is shrimp scampi made of? |
StrategyQA |
Do the anchors on Rede Globo speak Chinese? |
What languages do the anchors on Rede Globo speak? |
D.4 Baseline Prompts
For standard zero-shot and few-shot prompting of the baseline model, we formulate the prompt using the template in Table 16 with up to 1 exemplars.
Baseline few-shot Prompt |
You are an expert of world knowledge and physics. Your task is to solve the following question. Here are a few examples:
|
dataset |
Question |
Answer |
---|---|---|
MMLU Physics |
A point charge, Q1 = +1 mC, is fixed at the origin. How much work is required to move another charge, Q2 = +8 C, from the point (0, 4 meters) to the point (3 meters, 0)?
|
The work required to move a charge in an electric field is equal to the change in potential energy of the charge. The potential energy of a charge in an electric field is given by: U = k * Q1 * Q2 / r
where: k is the Coulomb constant, Q1 and Q2 are the charges, r is the distance between the charges.
In this case, the initial potential energy of Q2 is:
U1 = k * Q1 * Q2 / 4 meters
The final potential energy of Q2 is:
U2 = k * Q1 * Q2 / 3 meters
The change in potential energy is:
U = U2 - U1 = k * Q1 * Q2 * (1/3 meters - 1/4 meters)
Plugging in the values, we get: U = . The answer is 1) 6.0 J. |
MMLU Chemistry |
A sample of an unknown chloride compound was dissolved in water, and then titrated with excess Pb(NO3)2 to create a precipitate. After drying, it is determined there are 0.0050 mol of precipitate present. What mass of chloride is present in the original sample?
|
0.0050 moles of lead chloride precipitate contain 0.010 moles of chloride ions due to the 2:1 mole ratio between chloride ions and lead chloride.
The mass of 0.010 moles of chloride ions is 0.355 g.
Therefore, the mass of chloride present in the original sample is 0.355 g. |
GSM8K |
Bella bought stamps at the post office. Some of the stamps had a snowflake design, some had a truck design, and some had a rose design. Bella bought 11 snowflake stamps. She bought 9 more truck stamps than snowflake stamps, and 13 fewer rose stamps than truck stamps. How many stamps did Bella buy in all? |
The number of truck stamps is 11 + 9 = 20. The number of rose stamps is 20 - 13 = 7. Bella bought 11 + 20 + 7 = 38 stamps in all. |
TimeQA |
Who was the spouse of Anna Karina from 1968 to 1974? |
Anna Karina’s spouse from 1968 to 1974 was French actor Pierre Fabre. |
SituatedQA |
what is the biggest hotel in las vegas nv as of November 28, 1993 |
The biggest hotel in Las Vegas, Nevada as of November 28, 1993 was the Excalibur Hotel & Casino. |
MuSiQue |
What year saw the creation of the region where the county of Hertfordshire is located? |
1994 |
StrategyQA |
Would a Monoamine Oxidase candy bar cheer up a depressed friend? |
A Monoamine Oxidase (MAO) candy bar would not cheer up a depressed friend. MAO is an enzyme that breaks down neurotransmitters such as serotonin, dopamine, and norepinephrine. These neurotransmitters are associated with mood, so inhibiting MAO can lead to an increase in their levels. |
dataset |
Question |
Answer |
MMLU Physics |
A point charge, Q1 = +1 mC, is fixed at the origin. How much work is required to move another charge, Q2 = +8 C, from the point (0, 4 meters) to the point (3 meters, 0)?
|
Sure, let’s think step by step.
1. Calculate the electric potential energy of Q2 at the initial position. The electric potential energy (U) of a point charge Q2 at a distance r from another point charge Q1 is given by the following equation: U = k * Q1 * Q2 / r, where k is Coulomb’s constant.
Therefore, the electric potential energy of Q2 at the initial position is:
2. Calculate the electric potential energy of Q2 at the final position. The electric potential energy of Q2 at the final position is:
3. Calculate the work required to move Q2 from the initial position to the final position. The work (W) required to move a charge from one position to another is equal to the change in electric potential energy (U) between the two positions.
Therefore, the answer is 1) 6.0 J. |
dataset |
Question |
Answer |
MMLU Chemistry |
A sample of an unknown chloride compound was dissolved in water, and then titrated with excess Pb(NO3)2 to create a precipitate. After drying, it is determined there are 0.0050 mol of precipitate present. What mass of chloride is present in the original sample?
|
Here’s the step-by-step solution:
Determine the mole ratio: The balanced chemical equation for the reaction between a chloride compound (represented as MCl, where M is the unknown metal) and lead nitrate is: + +
Calculate the moles of chloride ions: Since 0.0050 moles of lead chloride precipitate were formed, there must have been 2 - 0.0050 = 0.010 moles of chloride ions in the original sample.
Calculate the mass of chloride ions: The molar mass of chlorine (Cl) is 35.45 g/mol. Therefore, the mass of 0.010 moles of chloride ions is:
Therefore, the mass of chloride present in the original sample is 0.355 g. The correct answer is option 1. |
dataset |
Question |
Answer |
GSM8K |
Bella bought stamps at the post office. Some of the stamps had a snowflake design, some had a truck design, and some had a rose design. Bella bought 11 snowflake stamps. She bought 9 more truck stamps than snowflake stamps, and 13 fewer rose stamps than truck stamps. How many stamps did Bella buy in all? |
Here are the steps to solve the problem:
Step 1: Find the number of truck stamps. Adding 9 to the number of snowflake stamps gives the number of truck stamps: 11 + 9 = 20
Step 2: Find the number of rose stamps. Subtracting 13 from the number of truck stamps gives the number of rose stamps: 20 - 13 = 7
Step 3: Add the number of snowflake, truck, and rose stamps to find the total number of stamps Bella bought: 11 + 20 + 7 = 38
Solution: Bella bought 38 stamps in all. |
D.5 Chain of Thought (CoT)
For zero-shot CoT prompting, we simply append Let’s think step by step. to the question to query the model.
For few-shot CoT prompting, we use the same template as the Baseline prompting in Sec. D.4 by replacing the few-shot examples using CoT responses, as shown in Tables 18, 19, 20, 21, and 22.
dataset |
Question |
Answer |
---|---|---|
TimeQA |
Who was the spouse of Anna Karina from 1968 to 1974? |
Step 1: Identify Anna Karina’s spouses:Jean-Luc Godard (1961–1965).
Step 2: Determine which spouse was married to Anna Karina from 1968 to 1974.
Therefore, Anna Karina’s spouse from 1968 to 1974 was Pierre Fabre. |
SituatedQA |
what is the biggest hotel in las vegas nv as of November 28, 1993 |
Sure, let’s think step by step: What were the largest hotels in Las Vegas in 1993? According to a 1993 article in the Las Vegas Review-Journal, the largest hotels in Las Vegas at the time were: MGM Grand (5,005 rooms) Excalibur (4,008 rooms) The Mirage (3,049 rooms) Flamingo Hilton (3,565 rooms) Caesars Palace (2,577 rooms) Which of these hotels was the largest on November 28, 1993? The MGM Grand opened in December 1993, so it would not have been the largest hotel on November 28, 1993. The Excalibur opened in June 1990, so it would have been the largest hotel on November 28, 1993. Therefore, the largest hotel in Las Vegas on November 28, 1993 was the Excalibur. |
dataset |
Question |
Answer |
---|---|---|
MuSiQue |
What year saw the creation of the region where the county of Hertfordshire is located? |
The county of Hertfordshire is in the ’East of England’ region. The ’East of England’ region was created in 1994. So, the answer is 1994. |
StrategyQA |
Would a Monoamine Oxidase candy bar cheer up a depressed friend? |
No, a Monoamine Oxidase (MAO) candy bar would not cheer up a depressed friend. In fact, it could be dangerous for them to consume. Here’s a step-by-step explanation: 1. MAO is an enzyme that breaks down neurotransmitters like serotonin, dopamine, and norepinephrine. These neurotransmitters are important for regulating mood, and low levels of them are associated with depression. 2. MAO inhibitors (MAOIs) are a type of medication that blocks the action of MAO, which can increase levels of these neurotransmitters in the brain. MAOIs are sometimes used to treat depression, but they can have serious side effects, including interactions with certain foods and medications. 3. A MAO candy bar would not contain enough MAOI to have a therapeutic effect on depression. In fact, it is unlikely that a candy bar could contain any MAOI at all, as these medications are typically taken in pill form. In conclusion, a MAO candy bar would not cheer up a depressed friend and could even be dangerous for them to consume. |
D.6 Take a Deep Breath (TDB)
We study the zero-shot prompting found in Yang et al. (2023): we take Take a deep breath and work on this problem step-by-step, and prepend it to the question.
Appendix E Examples of Error Analysis and Wins of Step-Back Prompting
E.1 MMLU error analysis
In Tables 23-27, we show one example for each of the 5 error categories we identified through error analysis on Step-Back Prompting.
Question |
An alarm whose frequency is 400 Hz is dropped out of a third-floor window. The student who drops it measures the frequency with a very sensitive oscilloscope. The measured frequency
|
---|---|
Principles |
Doppler Effect: the effect produced by the motion of a wave source, observer, or both that causes a change in wave frequency perceived by an observer relative to the frequency produced by the source. The Doppler effect is observed with all types of waves, including electromagnetic and acoustic waves. It was named after Austrian physicist Christian Doppler. |
Answer |
Using the Principles of Doppler Effect, we can solve the problem as following:
Step 1: Apply Doppler Effect to find out the measured frequency.
As the alarm falls, the distance between the alarm and the student decreases. This means that the frequency of the wave as measured by the student will increase. Therefore, the measured frequency will appear higher than 400 Hz, and the frequency will increase as the alarm falls.
Therefore, the correct answer is 0) appears higher than 400 Hz and the frequency increase as it falls. |
Question |
An astronaut lands on a planet whose mass and radius are each twice that of Earth. If the astronaut weighs 800 N on Earth, how much will he weigh on this planet?
|
---|---|
Principles |
Weight: W = m * g, where W is the weight, m is the mass and g is the acceleration due to gravity
|
Answer |
Using the Principles of Newton’s law of universal gravitation and Weight, we can solve the problem as following:
Step 1: Apply Newton’s law of universal gravitation to find out the acceleration due to gravity on the planet.
In this problem, we are given the following information:
Plugging these values into the formula, we get:
This is the same as the acceleration due to gravity on Earth.
Step 2: Apply Weight to find out the weight of the astronaut on the planet.
In this problem, we are given the following information:
Plugging these values into the formula, we get:
Therefore, the astronaut will weigh 800 N on the planet. The correct answer is 2) 800 N. |
Question |
Two spheres of net charge +5e and -6e briefly come into contact. Afterward, which of the following is a possible combination of net charges for the two spheres?
|
---|---|
Principles |
Law of conservation of charge: the total charge in a closed system remains constant.
|
Answer |
Using the Principles of Conservation of charge, we can solve the problem as following:
Step 1: Apply Conservation of charge to find out the possible combination of net charges for the two spheres.
Step 2: Find out the possible combination of net charges for the two spheres.
Therefore, the correct answer is 2) Both of the above. |
Question |
The circumference of a helium-filled balloon is measured for three different conditions: at room temperature, after being in a warm oven for 30 minutes, and after being in a freezer for 30 minutes. A student plotting the circumference cubed as a function of temperature T, should expect to find which of the following?
|
---|---|
Principles |
Ideal Gas Law: PV = nRT, where P is the pressure, V is the volume, n is the number of moles of gas, R is the ideal gas constant, T is the absolute temperature.
|
Answer |
Using the Principles of Charles’s Law and Gay-Lussac’s Law, we can solve the problem as following:
Step 2: Apply Gay-Lussac’s Law to find out the relationship between the pressure and temperature of the gas. Gay-Lussac’s Law states that the pressure of a fixed mass of a gas is directly proportional to the absolute temperature of the gas when volume is kept constant. This means that if the temperature of the gas increases, the pressure of the gas will also increase.
Step 3: Apply Charles’s Law and Gay-Lussac’s Law to find out the relationship between the circumference and temperature of the gas.The circumference of the balloon is directly proportional to the volume of the gas. This is because the circumference of the balloon is the distance around the outside of the balloon, and the volume of the gas is the amount of space that the gas takes up.
The pressure of the gas is directly proportional to the temperature of the gas. This is because the pressure of the gas is the force that the gas exerts on the walls of the balloon, and the temperature of the gas is the measure of the average kinetic energy of the gas molecules.
Therefore, the circumference of the balloon is directly proportional to the temperature of the gas. This means that if the temperature of the gas increases, the circumference of the balloon will also increase.
Step 4: Apply the relationship between the circumference and temperature of the gas to find out the relationship between the circumference cubed and temperature of the gas. The circumference cubed is directly proportional to the temperature of the gas. This is because the circumference cubed is the volume of the balloon, and the volume of the balloon is directly proportional to the temperature of the gas.
Therefore, the student should expect to find a cubic relationship between and T. The correct answer is 0) a cubic relationship between and T. |
Question |
Physics students are checking the constant acceleration equations of kinematics by measuring the velocity of a tennis ball that is dropped and falls 6 meters and then passes through a photogate. The predicted velocity is above the velocity measured by the photogate. Which of the following best describes the cause of the large percent difference?
|
---|---|
Principles |
Vertical motion: y = 0.5 * g * , where y is the vertical displacement, g is the acceleration due to gravity, t is the time.
|
Answer |
Using the Principles of Vertical motion and Horizontal motion, we can solve the problem as following:
Step 1: Apply the Vertical motion equation to find the time it takes for the ball to fall 6 meters.
Step 2: Apply the Horizontal motion equation to find the velocity of the ball when it passes through the photogate.
Step 3: Compare the predicted velocity with the velocity measured by the photogate. The predicted velocity is above the velocity measured by the photogate, so the predicted velocity is 1.2 * 5.36 m/s = 6.43 m/s. Step 4: Find the cause of the large percent difference. The predicted velocity is 6.43 m/s, but the velocity measured by the photogate is 5.36 m/s. The difference is 1.07 m/s. This difference is caused by air resistance. Air resistance is a force that opposes the motion of an object through the air. It is caused by the friction between the object and the air molecules. The greater the surface area of the object, the greater the air resistance. The tennis ball has a large surface area, so it experiences a lot of air resistance. This air resistance slows down the tennis ball and causes the difference between the predicted velocity and the velocity measured by the photogate. Therefore, the correct answer is 0) The ball changes its shape while falling. |
E.2 Example Wins from Step-Back Prompting
Tables 28- 30, 31 32, 33 and 34 illustrate the some successful examples of Step-Back Prompting on MMLU-Physics, MMLU-Chemistry, TimeQA, SituatedQA, and StrategyQA respectively.
dataset |
example |
MMLU-Physics |
Original Question: What happens to the pressure, P, of an ideal gas if the temperature is increased by a factor of 2 and the volume is increased by a factor of 8 ?
Step 2: Apply Ideal gas law to find out the pressure of an ideal gas when the temperature is increased by a factor of 2 and the volume is increased by a factor of 8.
If the temperature is increased by a factor of 2, then T becomes 2T. If the volume is increased by a factor of 8, then V becomes 8V.
Substituting these values into the ideal gas law, we get:
P(8V) = nR(2T)
Dividing both sides by 8V, we get:
P = nR(2T) / 8V
We can see that the pressure has decreased by a factor of 4.
Therefore, the correct answer is 1) P decreases by a factor of 4. |
dataset |
example |
MMLU-Physics |
Original Question: A liquid flows at a constant flow rate through a pipe with circular cross-sections of varying diameters. At one point in the pipe, the diameter is 2 cm and the flow speed is 18 m/s. What is the flow speed at another point in this pipe, where the diameter is 3 cm?
|
dataset |
example |
MMLU-Physics |
Original Question: A 2 F capacitor is connected directly to a battery. When the capacitor is fully charged, it stores 600 C of charge. An experimenter replaces the 2 F capacitor with three 18 F capacitors in series connected to the same battery. Once the capacitors are fully charged, what charge is stored on each capacitor?
Step 2: Apply Capacitors in series to find out the equivalent capacitance of the three 18 F capacitors in series.
|
dataset |
example |
---|---|
MMLU-Chemistry |
Original Question: Which of the following indicates that a reaction is spontaneous?
The correct answer is 0) at equilibrium there are more products than reactants. |
MMLU-Chemistry |
Original Question: Potassium-40 is a minor isotope found in naturally occurring potassium. It is radioactive and can be detected on simple radiation counters. How many protons, neutrons, and electrons does potassium-40 have when it is part of K2SO4?
|
dataset |
example |
---|---|
TimeQA |
Original Question: Carlos Ascues played for which team from 2013 to 2014?
|
TimeQA |
Original Question: Seth Nana Twumasi played for which team between Jan 2011 and Feb 2011?
|
TimeQA |
Original Question: What position did Carmel Hanna take in May 2002?
|
dataset |
example |
---|---|
SituatedQA |
Original Question: when was the last time miss texas won miss america as of November 29, 2010
|
SituatedQA |
Original Question: when is the last time congress passed a complete budget as of 2021
|
SituatedQA |
Original Question: when was the last time unc did not make the ncaa tournament as of 2003
|
dataset |
example |
---|---|
StrategyQA |
Original Question: Could you drive a Rowe 550 to the 2008 Summer Olympics?
|
StrategyQA |
Original Question: what channel does thursday night football come on regular tv as of 2017?
|
StrategyQA |
Original Question: who is the present president of assam sahitya sabha as of 2018?
|