Framing experts' (dis)agreements about uncertain environmental events

Received: 21 August 2018 Revised: 15 April 2019 Accepted: 15 April 2019 DOI: 10.1002/bdm.2132 RESEARCH ARTICLE Framing experts' (dis)agreements about uncertain environmental events Erik Løhre1,2 | Agata Sobkow3 | Sigrid Møyner Hohle1 | Karl Halvor Teigen1,4 1 Software Engineering Department, Simula Research Laboratory, Oslo, Norway Abstract 2 Agreements and disagreements between expert statements influence lay people's Department of Psychology, Inland Norway University of Applied Sciences, Lillehammer, Norway 3 Wroclaw Faculty of Psychology, Center for Research on Improving Decision Making (CRIDM), SWPS University of Social Sciences and Humanities, Wroclaw, Poland 4 Department of Psychology, University of Oslo, Oslo, Norway Correspondence Erik Løhre, Høgskolen i Innlandet, Postboks 400, 2418 Elverum, Norway. Email: erik.lohre@gmail.com Funding information SWPS University of Social Sciences and Humanities, Grant/Award Number: BST/ Wroc/2017/A/08; Norges Forskningsråd, Grant/Award Number: 235585/E10 beliefs. But few studies have examined what is perceived as a disagreement. We report six experiments where people rated agreement between pairs of probabilistic statements about environmental events, attributed to two different experts or to the same expert at two different points in time. The statements differed in frame, by focusing on complementary outcomes (45% probability that smog will have negative health effects vs. 55% probability that it will not have such effects), in probability level (45% vs. 55% probability of negative effects), or in both respects. Opposite frames strengthened disagreement when combined with different probability levels. Approximate probabilities can be “framed” in yet another way by indicating reference values they are “over” or “under”. Statements that use different directional verbal terms (over vs. under 50%) indicated greater disagreement than statements with the same directional term but different probability levels (over 50% vs. over 70%). Framing and directional terms similarly affected consistency judgments when both statements were issued by the same expert at different occasions. The effect of framing on perceived agreement was significant for medium (10 and 20 percentage points) differences between probabilities, whereas the effect of directional term was stable for numerical differences up to 40 percentage points. To emphasize agreement between different estimates, they should be framed in the same way. To accentuate disagreements or changes of opinion, opposite framings should be used. K E Y W OR D S directional verbal terms, disagreement, experts, framing, probability estimates 1 | I N T RO D U CT I O N over time, making it less obvious which expert opinion to follow. It is therefore no surprise that agreements and disagreements among Lay people often rely on expert opinions when making decisions experts are an important concern for the public. about a variety of topics, ranging from health (which foods should When people are informed about the expert consensus regarding be included in my diet? and will air pollution influence my health?) climate change (i.e., that 97% of climate scientists agree that CO2 emis- and finance (which home insurance should I choose?) to complex sions cause global warming), their acceptance of man‐made climate global issues such as climate change (how can it be mitigated?). change increases (Lewandowsky, Gignac, & Vaughan, 2013; van der Lin- However, experts do not always agree and may change their mind den, Leiserowitz, Feinberg, & Maibach, 2015), whereas perceived J Behav Dec Making. 2019;1–15. wileyonlinelibrary.com/journal/bdm © 2019 John Wiley & Sons, Ltd. 1 LØHRE 2 ET AL. dissent is associated with decreased support for environmental policies change is occurring, one could also state that 3% of climate scientists (Aklin & Urpelainen, 2014).1 Nagler (2014) found that conflicting do not agree with this. Different, logically equivalent ways of express- research findings concerning health benefits and risks of different food ing the same information is known as framing, and a framing effect products made people confused about what to eat and could even make occurs when recipients of the information respond differently depend- people start doubting nutrition advice more generally. Groups with an ing on the way the information is expressed (Teigen, 2016). The classic interest in undermining the public's belief in a body of scientific evi- example of a framing effect is the so‐called Asian disease problem dence seem aware of the persuasive power of consensus and have (Tversky & Kahneman, 1981), which showed that people's preferences actively used “experts” who contradict the best current scientific under- for “safe” and “risky” options depended on whether the outcomes standing to spread doubt, for instance, claiming that there is uncertainty were described as gains (200 out of 600 people will be saved) or as about whether smoking causes lung cancer or whether climate change is losses (400 out of 600 people will die). Another well‐known example caused by increased CO2 emissions (Oreskes & Conway, 2010). Simi- is that a yogurt described as “95% fat free” is perceived as healthier larly, research on conflict aversion indicates that people dislike disagree- than a yogurt described as containing “5% fat” (Sanford, Fay, Stewart, ment. For instance, people find two eyewitnesses less reliable when & Moxey, 2002). they give conflicting testimony about a car used in a robbery (one wit- Framing effects have been explained in many different ways, but ness thought the car was blue and the other thought it was green) than for the present purpose, the “information leakage” perspective may when they agree that the facts are ambiguous (both witnesses state that be particularly useful. Sher and McKenzie (2006) proposed that the car was “either blue or green”; Smithson, 1999). speakers usually do not choose which frame they use at random and Agreement is most often conceived as similarities in opinion that listeners are aware of this. Thus, although two different frames between different experts, but experts could also agree more or less are logically equivalent, they may “leak information” about the state with themselves, by remaining consistent or changing their opinion of the world or about the speaker's attitudes towards the object of over time. Although there is less research on this topic, it has been discussion. For instance, glasses that are in the process of being filled proposed that people prefer consistency, both from themselves and are more often described as “half full,” whereas glasses that were pre- from others (Tedeschi, Schlenker, & Bonoma, 1971). In line with this, viously full and are in the process of being emptied are more often people infer that a person who gives more consistent estimates of described as “half empty” (McKenzie & Nelson, 2003). This analysis an uncertain quantity over time is more skilled than someone who of framing effects indicates that speakers who choose different frames changes his or her estimate (Falk & Zimmermann, 2017) and believe also have different opinions about what people should attend to politicians should not change their mind—unless of course they end and/or about the state of the world; in other words, there are issues up agreeing with the participant's own viewpoint (Croco, 2016). on which they disagree. In summary, previous research suggests that people prefer agree- In a financial version of the Asian disease problem, Teigen and ment between different experts and consistency from one expert over Nikolaisen (2009) found that people perceived two brokers who pre- time. However, to our knowledge, it has not previously been investi- sented the options in the dilemma as either losses or gains to be giving gated under which circumstances lay people will perceive experts as different recommendations. The broker using a gain frame was disagreeing or as changing their mind. How similar must two state- thought to favor the safe option, whereas the broker using a loss ments be to be regarded as supporting each other, and how different frame was believed to recommend the risky option. Thus, different should they be to be regarded as incompatible? Given the ubiquity of frames indeed indicated different opinions about the options. Simi- probabilistic statements in expert opinions, particularly in areas char- larly, Sher and McKenzie (2008) found that participants' preferences acterized by uncertainty, such as climate change and environmental or goals influenced how they framed the options in the Asian disease risks, the present studies focus on numerical probability estimates. problem to others, again suggesting that different opinions may be Specifically, we provide evidence that the choice of frames and direc- reflected by choice of frames. Framing may also influence perceptions tional verbal terms in the communication of numerical probability esti- of agreement between predictions and outcomes, with predictions mates may influence the degree to which two experts are seen to seen as more accurate when the frame of the prediction was congru- disagree and the degree to which one expert is seen to change his ent with the outcome (Yeung, 2014). For instance, the prediction “Uni- or her mind. By this research, we make a first step in identifying com- versity B has a 30% chance of winning” was seen as more accurate municative factors that contribute to perceived (dis)agreement. when University B won the football game than the logically interchangeable prediction that “University A has a 70% chance of 1.1 | Framing and agreement A numerical estimate may be expressed in several different ways. For instance, if 97% of climate scientists agree that human‐caused climate 1 Ironically, there is some disagreement about whether consensus plays a key role in convincing people about contested facts like climate change (Kahan, 2008; van der Linden, Leiserowitz, & Maibach, 2016). Nevertheless, in our opinion, the weight of the evidence suggests that expert consensus can be an influential cue for lay people. winning”. Building on these results, we hypothesized that probability statements within the same frame will be seen as agreeing more than probability statements with opposite frames (i.e., probability statements focusing on opposite, complementary outcomes). Similarly, an expert who changes his or her numerical probability estimate over time but keeps the frame constant may be seen as more consistent than an expert who changes the way his or her probability estimate is framed. LØHRE ET AL. 3 This hypothesis could also be seen as consistent with fuzzy‐trace the- findings illustrate that these terms are directional, pointing either ory (e.g., Reyna, 2004; Reyna & Brainerd, 1991), which proposes that upwards (“over 30%”) or downwards (“under 50%”). people generally rely on gist rather than verbatim information in rea- Terms like “over” and “under” may be used to communicate soning. In other words, people would be more concerned with the quantities other than probabilities, for instance, prices. Listeners qualitative difference between frames (the gist) rather than quantita- believed that a speaker who states that a product costs “over 600” tive differences within frames (the verbatim numbers). found the product expensive and advised against buying it, whereas Although our hypothesis is consistent with theoretical approaches a speaker giving an estimate of “under 900” was thought to find the that emphasize pragmatic aspects of framing effects, like information product cheap and recommend purchase (Teigen, 2008). Speakers leakage and fuzzy‐trace theory, approaches that see framing effects seem to be aware of this, as they overwhelmingly chose upper as demonstrations of irrational behavior (Tversky & Kahneman, bound statements (“less than/maximum 900”) when they wanted 1986) would not necessarily make the same prediction. From a dual‐ to encourage a purchase and lower bound statements (“more process perspective, framing effects occur due to intuitive Type 1 pro- than/minimum 600”) when they wanted to discourage it (Teigen, cesses, which may be overridden by more controlled Type 2 pro- Halberg, & Fostervold, 2007). cesses, depending on the circumstances (Kahneman, 2011). In our This brief summary indicates that single‐bound probability state- studies, we employed a within‐subjects design, with participants ments (e.g., “more than 30%” or “less than 50%”), which combine a directly comparing statements given in opposite frames. This design directional verbal modifier with a numerical probability, may reveal has been claimed to allow participants to identify the variable of differences in opinion, similar to the use of different frames. interest and adjust their responses accordingly (Kahneman & Freder- Although studies of framing normally use logically equivalent state- ick, 2005).2 In other words, simultaneous presentation of both frames ments (e.g., a 10% mortality rate is logically equivalent to a 90% sur- should make the logical equivalence of the statements transparent, vival rate), this is not necessarily the case for directional terms (e.g., and participants could consequently rate statements in opposite “over 30%” is not logically equivalent to “under 50%”). However, frames as (dis)agreeing to the same extent as equivalent statements framing can be defined more broadly: “To frame is to select some in the same frame. Hence, evidence supporting our hypothesis aspects of a perceived reality and make them more salient in a com- would be more consistent with pragmatic than with dual‐process municating text, in such a way as to promote a particular problem approaches. definition” (Entman, 1993, p. 52). Thus, even though directional terms and logically equivalent frames differ, they may have many of the same communicative functions and effects. Hence, we hypothesize that approximate probabilistic statements using different 1.2 | Approximate probabilities and directional verbal terms directional terms (e.g., “over 50%” vs. “under 50%”) will indicate more disagreement than statements with the same direction but different numbers (e.g., “over 50%” vs. “over 70%”). Experts do not always give probability estimates as a precise number. If the exact probability is uncertain, an expert may state that the probability is “between 50 and 70%,” or she he might simplify and say that the probability is “more than 50%” or “less than 70%”. This choice to 1.3 | The present research focus on either the lower or upper boundary of a probability range We conducted six experiments to investigate how choices in uncer- might seem entirely arbitrary. However, much in the same way as tainty communication may influence lay people's perceptions of different frames may convey different opinions, recommendations, expert (dis)agreement. Participants in all experiments read expert or reference points, directional terms like “more than”, “less than”, statements about different environmental issues (e.g., air pollution “over”, and “under” can carry implicit information about a speaker's and wind park productivity), containing numerical probabilities preferences or beliefs in addition to the quantitative information expressed in different ways. (Teigen, 2008). In Experiment 1a, probability statements from two different Imagine, for instance, two experts stating that the probability of a experts were expressed using either the same frame (45% vs. 55% glacier shrinking to half its size is either “over 30%” or “under 50%”. probability that the air pollution in a city will lead to negative effects Although participants in a recent study believed the implied point on the health of the citizens) or opposite frames (45% probability that probability of the first expert was lower (36%) than that of the second the air pollution will lead to negative health effects vs. 45% probability expert (48%), they nevertheless reported that experts using “over” that it will not have such effects). Participants were asked to rate the were more inclined to expect the event to occur (Hohle & Teigen, perceived (dis)agreement between the experts, as well as their trust 2018a). The same study also showed that an expert stating that the in research on smog and the perceived risk of air pollution. Experiment probability is “over 30%” was thought to have made a lower estimate 1b used the same design, but expert statements were here attributed last year, whereas “under 50%” indicated a decreasing trend. Both to the same expert at two points in time. 2 However, Aczel, Szollosi, and Bago (2013) found framing effects using a within‐subjects design. In Experiment 2a, participants received probability statements that differed either in the directional term used (“over 50%” vs. “under LØHRE 4 ET AL. 50%”) or in their probability level (“over 50%” vs. “over 70%”) and influence citizens' health. Observe that both statements imply the were asked to rate between‐expert agreement for one scenario and same actual probability that smog will be harmful (55%), as the second within‐expert agreement (consistency) for another scenario. Experi- statement is logically complementary to the first one. Hence, the ment 2b repeated this design but switched the comparisons made experts agreed in terms of actual probability about the outcome, but between subject and within subject. they disagreed in the way this probability is framed. Experiment 3 investigated whether the effects of framing and In Condition B (the opposite‐framing condition), both experts gave directional terms were robust with a larger range of differences the same numerical probability (45%) but related to different out- between the numerical probability estimates. One scenario involved comes (Expert 1: will influence and Expert 2: will not). In other words, statements given in the same versus opposite frames, and another the experts disagreed in terms of both frame and actual probability scenario involved statements featuring same versus opposite direc- that air pollution will be harmful, whereas the nominal probabilities tional terms. were the same. While participants in the above experiments were asked to rate In Condition C (the same‐framing condition), both experts the perceived (dis)agreement between probabilistic statements, dis- expressed their estimates with respect to the same outcome but agreement was used as the independent variable in Experiment 4. Par- disagreed about the actual probabilities involved (Expert 1: 45% and ticipants received one expert statement (a 55% probability that a solar Expert 2: 55% probability that smog will negatively influence citizens' power facility will be successful in covering energy demands) and were health). Thus, the predictions in Conditions B and C are formally equiv- asked to evaluate what an expert who disagreed (or agreed) would alent to each other but framed in complementary ways. These condi- say. Would they frame their statements in terms of probabilities of tions describe two experts who disagreed with the same amount (10 success (45% probability it will be successful) or failure (55% probabil- percentage points), in contrast to the experts in Condition A who, ity it will not be successful)? from a formal point of view, did not disagree at all. After reading the expert statements, participants were asked to answer two questions related to perceived agreement (e.g., to what | 2 EXPERIMENT 1A extent do these experts agree?), two questions about the trustworthiness of research about the smog (e.g., based on these statements, to The aim of Experiment 1a was to demonstrate how the framing of a what extent does the research on smog seem trustworthy?), and two probabilistic prediction (e.g., a 55% probability that outcome X will questions measuring the perception of smog as a risk (e.g., based on occur vs. a 45% probability that outcome X will not occur) could influ- these statements, do you think smog is a real threat?). Participants ence perceptions of expert disagreement and risk assessment. Specif- responded using Likert scales ranging from 1 (definitely not) to 6 (defi- ically, we hypothesized that expert statements with a difference in nitely yes). As the two questions for each subscale were significantly numerical probability expressed in the same frame would be perceived related, r(148) = .77, p < .001, for expert agreement; r(148) = .79, as agreeing more than the corresponding statements expressed in p < .001, for trustworthiness; r(148) = .79, p < .001, for perceived risk, opposite frames. an average for each scale was computed and used in the analysis. Our main interest here was the ratings of agreement, and we hypothesized 2.1 | Method lower perceived agreement between experts in the opposite‐framing condition (B) than in the same‐framing condition (C), whereas we 2.1.1 | Participants had no clear prediction for Condition A. The ratings of trust and risk were included for exploratory purposes. One hundred fifty students at a Polish university (median age 22 years; 114 women and 29 men, and seven did not indicate gender) participated in a study in exchange for course credits. Participants were ran- 2.2 | Results domly assigned to one of three experimental conditions. Three one‐way analyses of variance (ANOVAs) with the average 2.1.2 | Materials and procedure In each condition, participants were shown a brief vignette containing statements from two experts about how smog in Wroclaw could influence the health of the citizens. The statements differed in two ways, by probability estimate (45% or 55%) and by outcome (the smog “will” or “will not” influence health). In Condition A (the “logical equivalence” condition), the first expert indicated that there is a 55% probability that breathing air in Wroclaw will negatively influence citizens' health, whereas the second expert indicated that there is a 45% probability the smog will not negatively responses to the questions about expert agreement, trust, and perceived risk as dependent variables were performed (see Table 1 for summary of results). Experimental condition significantly influenced expert agreement, F (2, 147) = 21.341, p < .001, η2p = .225.3 Participants rated expert agreement as lowest in Condition B, where the experts disagreed both in terms of the framing of their prediction and in terms of the actual probability that the smog will be harmful. 3 Levene's test indicated that the variance across groups was unequal, F (2, 147) = 9.218, p < .001. However, the largest variance was less than four times larger than the smallest one (the ratio was 2.46), which, according to Howell (2006, p. 344), indicates that the analysis of variance can still be used. We follow this guideline throughout the current paper. Using nonparametric analysis where possible yielded similar results. LØHRE ET AL. 5 TABLE 1 Mean ratings (1–6, higher numbers indicate more agreement, trust, and risk) of probabilistic messages coming from different experts (Experiment 1a) or the same expert at different occasions (Experiment 1b) Experiment DV Logical equivalence (Condition A) M (SD) Opposite framing (Condition B) M (SD) Same framing (Condition C) M (SD) Experiment 1a (two experts) Agreement Trust Perceived risk 3.78 (1.68) 2.96 (1.26) 3.14 (1.10) 2.05 (1.36) 2.12 (1.26) 2.79 (1.18) 3.40 (1.07) 3.20 (1.15) 3.78 (1.19) Experiment 1b (same expert) Consistency Trust Perceived risk 5.05 (1.33) 3.57 (1.42) 3.97 (1.44) 3.40 (1.80) 3.04 (1.36) 3.46 (1.27) 4.16 (1.03) 3.46 (1.25) 3.88 (1.30) Note. Condition A (45% that smog will be harmful vs. 55% that smog will not be harmful). Condition B (45% that smog will be harmful vs. 45% that smog will not be harmful). Condition C (45% vs. 55% that smog will be harmful). Abbreviation: DV, dependent variable. Post hoc tests (Bonferroni) showed that agreement was rated as sig- be that these participants conceptualize agreement as “logical equiva- nificantly lower in Condition B than in Conditions A and C (ps < .001), lence”. However, if this were the case, we would expect the ratings in whereas Conditions A and C did not differ significantly from each Condition A (M = 3.78), to be closer to the end point of the agreement other (p = .523). For the main comparison of Conditions B and C scale (6.00) than to its midpoint (3.5). Thus, even the ratings in Condi- (opposite framing vs. same framing), the effect size was large, Cohen's tion A indicate that statements in opposite frames reduce perceived d = 1.10. agreement. As an additional illustration of the results, the expert agreement ratings were split by the midpoint, with ratings up to 3.5 indicating perceived disagreement and ratings above 3.5 indicating perceived | 3 E X P E R I M E N T 1B agreement. The percentage of participants giving ratings below the midpoint was higher in Condition B (80%) than in Conditions C The aim of Experiment 1b was to test whether the framing effects (58%) and A (46%). observed in Experiment 1a are replicated when participants are told A similar pattern of results was observed for trust, F(2, that the statements are provided by the same expert at different = .145. People rated the trustworthiness points in time. We hypothesized that experts changing frames would of research as significantly lower in Condition B than in Conditions A be rated as less consistent than experts making comparable predic- and C, ps < .001, whereas the difference between Conditions A and C tions within the same frame. 147) = 12.493, p < .001, η2p was not significant (p = .876). Moreover, experimental condition significantly influenced per- 3.1 | Method ceived risk, F (2, 147) = 9.420, p < .001, η2p = .114. In this case, however, the perception of smog as a risk was rated as highest in 3.1.1 | Participants Condition C, which differed significantly from both Condition A, p = .019, and Condition B, p < .001. The difference between A and Participants were 121 students (median age = 26; 95 women and 26 B was not significant (p = .397). men) recruited via a university research pool panel (Wroclaw, Poland) in exchange for course credits. They were randomly assigned to one of three conditions of an online questionnaire. 2.3 | Discussion 3.1.2 | Materials and procedure Experiment 1a aimed to investigate whether framing influences perceived agreement between experts. The results gave two seemingly In each condition, participants were shown a brief vignette containing contradictory answers: Rated agreement was highest in Condition A, the same two statements as in Experiment 1a regarding the influence where the same prediction was framed in two different ways, suggest- of smog on citizens' health, but the statements were presented as ing that frame was not crucially important. However, a comparison of stemming from one expert at two points in time, with 1 year between Conditions B and C, which are logically equivalent to each other, the two statements. The two statements were varied with regard to showed that experts who make different predictions are perceived probability level and frame in the same way as in Experiment 1a. to disagree even more when using opposite frames. It might be that After reading the vignette, the participants received two questions participants in the different conditions conceptualize agreement in dif- about the perceived consistency of the expert, two questions about ferent ways. For instance, because the logical equivalence of the the reliability of smog research, and two questions about the percep- statements is arguably salient for participants in Condition A, it might tion of smog as a risk, using the same six‐point scales as in Experiment LØHRE 6 ET AL. 1a. The two questions for each subscale were significantly related, more naturally contrasted and interpreted as disagreement, in line r(119) = .56, p < .01, for expert consistency; r(119) = .88, p < .001, with theories about similarity and dissimilarity testing in comparison for trustworthiness; r(119) = .71, p < .001, for perceived risk, and the processes (Mussweiler, 2003). averages for each scale were used in the subsequent analyses. As in Experiment 1a, our main interest was in the ratings of agreement (consistency), whereas ratings of trust and risk were included for exploratory purposes. 4 | E X P E R I M E N T 2A While Experiments 1a and 1b framed predictions as chances for a target outcome to occur or not occur, they were in the two next exper- 3.2 | Results iments “framed” in another way, namely, as being “over” or “under” a stated probability percentage. Such single‐bound probability estimates Three 1‐way ANOVAs with consistency, trust, and perceived risk as dependent variables were performed (see Table 1 for summary of results). Experimental condition significantly influenced consistency, F (2, 118) = 12.81, p < .001, η2p = .18. The participants rated expert consistency as lowest in Condition B, where the expert changed both the frame and the probability level of harm. Condition B differed significantly from each of the two other conditions (ps < .001). There was also a significant difference between Condition A and Condition C (p = .012), with the highest ratings of consistency given to Condition A, where the expert had changed the frame but retained the probability of harm. For the comparison of main interest (Condition B: opposite framing vs. Condition C: same framing), the effect size was medium, Cohen's d = 0.52. The percentage of participants giving ratings below the midpoint of 3.5 (i.e., indicating that the expert had changed his or her opinion) was highest in Condition B (55.6%) followed by Conditions C (36.7%) and A (19.4%). In this experiment, condition influenced neither trust nor risk perception ratings, ps > .19. can differ with respect to both probability level and directional term (type of bound). Experiment 2a explored the importance of these two characteristics for judgments of agreement or disagreement between experts. Our hypothesis was that two forecasts with different probability level, but using the same directional term, would be perceived to be more in agreement than forecasts that use different directional terms. 4.1 4.1.1 | Method | Participants Participants were 106 students recruited in breaks between lectures at a Norwegian university (71 women and 26 men, nine did not indicate gender; median age 23 years). They were randomly allocated to two conditions by being given two different short questionnaires. 4.1.2 | Materials and procedure The questionnaires contained two scenarios, one about (dis)agreement 3.3 | Discussion between experts and the other about revised estimates from one expert (consistency). The results of Experiment 1b largely replicated the findings from Experiment 1a for one expert's consistency over time. Again, the rated agreement (consistency) was highest in Condition A, where the two statements are logically equivalent to each other, and was lowest in Condition B, where there is a difference in actual probability as well as in frame, with Condition C (difference in actual probability but same frame) in between. Interestingly, differences within and between experts were judged differently. Overall, ratings of within‐expert agreement (consistency) were higher (Experiment 1b: M = 4.2, SD = 1.5) than ratings of between‐expert agreement (Experiment 1a: M = 3.1, SD = 1.6)4. As the experiments were performed on different samples and at different times, no strong conclusions can be drawn, but one possible interpretation of this unexpected finding could be that although disagreements between different people are commonplace, people are 4.1.3 | Agreement between experts The first scenario briefly described two pairs of experts who were asked to evaluate the prospect of oil drilling outside Lofoten in the North Sea (a proposed but controversial location) before the year 2020. In Condition A, a first pair of experts stated the chances for such an activity were “over 50%” (Expert 1) or “under 50%” (Expert 2), whereas a second pair of experts thought the probabilities were “over 50%” or “over 70%”. In Condition B, the chances were stated as “over 70%” or “under 70%” by the first pair of experts, and “under 50%” vs. “under 70%” by the second pair of experts. Thus, in both conditions, the second pair disagreed with respect to probability level but used the same type of bound (i.e., the same directional term). The first pair used opposite bounds but disagreed perhaps less about probabil- generally expected to be consistent over time. Hence, even if an ity level.5 Table 2 (upper half) gives an overview over which state- expert has changed her or his estimate or her or his framing, she or ments were presented to participants in Conditions A and B. he is still assumed to be roughly in agreement with herself or himself Participants were asked to rate both pairs of experts on Likert scales (assimilation), whereas a similar difference between two experts is 5 In Study 1 in Hohle and Teigen (1981), participants thought, on average, that “over 50% 4 Similar results were found for trust and risk ratings. probability” meant 52.4% and that “under 50% probability” meant 48.1%. LØHRE ET AL. 7 TABLE 2 Mean ratings of agreement (1–5, higher numbers indicate more agreement) between different single‐bound probability estimates in Experiment 2a (within‐subjects comparisons) and Experiment 2b (between‐subjects comparisons) Pairs with different direction M (SD) Pairs with same direction M (SD) t (df) p Cohen's d Over 50% vs. under 50% chanceA 2.30 (0.98) Over 50% vs. over 70% chanceA 3.39 (0.76) 6.01 (53) <.001 0.82 Over 70% vs. under 70% chanceB 2.24 (0.99) Under 70% vs. under 50% chanceB 3.04 (0.77) 4.43 (50) <.001 0.76 Over 50% chance vs. under 50% chanceA 2.86 (0.96) Over 50% vs. over 70% chanceB 3.52 (0.78) 5.69 (228) <.001 0.75 Over 70% vs. under 70% chanceA 2.60 (0.96) Under 70% vs. under 50% chanceB 3.13 (0.86) 4.44 (223.43) <.001 0.58 Experiment 2a Experiment 2b Note. Statements marked with letters A and B as subscript were presented to participants in Conditions A and B, respectively. according to how much the experts within each pair appeared to agree with each other, from 1 (disagree completely) to 5 (agree completely). Experts with the same directional term were more in agreement. An expert saying “over 50%” and another saying “over 70%” chance for an increase in sea level were perceived to agree more than they 4.1.4 | disagree (26 vs. 6 scores above the scale midpoint in Condition A), Change of opinions whereas participants were more split in their ratings of experts saying The second scenario described two experts who assessed the chances “under 50%” and “under 70%” (16 vs. 14 scores above the midpoint in of a proposed project of renewable electricity from a windmill park in Condition B). In both conditions, the same‐direction pair agreed more Western Norway. In Condition A, Expert 1 said last year that the than the different‐direction pair (Table 2). chances were “over 50%” for the project to succeed; now she says the chances are “under 50%”. Expert 2 said last year “over 50%” chance; now he says “over 30%”. In Condition B, Expert 1 had changed 4.2.2 | Change of opinions from “over 30%” to “under 30%”, whereas Expert 2 had changed from “under 30%” to “under 50%” chance. Table 3 (upper half) gives an In the second scenario, participants were introduced to two overview over which statements were presented to participants in experts who had changed their predictions of the feasibility of a Conditions A and B. The experts were rated for changed opinions, projected windmill park. The first expert had changed her predictions from 1 (means the same as before) to 5 (means something completely from over to under 50% (in Condition A) or from over to under 30% different). (in Condition B). She was in both cases rated above the midpoint on the scale for changed opinions and was judged to have changed her 4.2 | opinion more than the second expert, who had become 20 percentage Results points more certain (in Condition A) or 20 points less certain (in Con- 4.2.1 | dition B) but had retained the directionality of the original prediction. Agreement between experts Mean ratings of changed opinions are shown in the upper half of Experts in the different‐direction pairs, whose statements differed by Table 3. being over versus under 50% (over vs. under 70%) were perceived to To conclude, we found in both scenarios a tendency to regard two disagree more than they agreed (see upper left panel of Table 2). In probability estimates with opposite directional terms (over vs. under) as Condition A, there were 37 agreement ratings below the scale mid- more different than two estimates with the same directional term but point of 3.0, versus 8 ratings above; in Condition B, 31 below versus a larger numerical difference. Observe that the meaning of high scores 6 above. in the two response scales is different. In Scenario 1, high scores mean TABLE 3 Mean ratings of opinion change (1–5, higher numbers indicate more opinion change) in experts who have revised their single‐bound probability estimates by changing direction and without changing direction in Experiment 2a (within‐subjects comparisons) and Experiment 2b (between‐subjects comparisons) Changes of direction M (SD) Changes with same direction M (SD) t (df) p Cohen's d From over 50% to under 50% chanceA 3.70 (1.18) From over 50% to over 30% chanceA 3.20 (0.90) 2.53 (53) .014 0.33 From over 30% to under 30% chanceB 3.67 (1.28) From under 30% to under 50% chanceB 3.02 (1.01) 2.62 (50) .012 0.44 From over 50% to under 50% chanceA 3.56 (1.27) From over 50% to over 30% chanceB 3.07 (1.02) 3.22 (222.53) .001 0.43 From over 30% to under 30% chanceA 3.52 (1.25) From under 30% to under 50% chanceB 3.24 (1.01) 1.86 (222.72) .064 0.25 Experiment 2a Experiment 2b Note. Statements marked with letters A and B as subscript were presented to participants in Conditions A and B, respectively. LØHRE 8 ET AL. that the two experts agree; in Scenario 2, a high score means that the saying “over 50%” and “over 70%” were in better agreement than expert's opinion has changed. those saying “under 70%” and “under 50%”, t(117) = 5.07, p < .001 (repeated measures test in Condition B). 5 | EXPERIMENT 2B 5.2.2 In Experiment 2a, expert pairs with same directional terms were compared within subjects to expert pairs using opposite directional terms. This experiment was a replication of Experiment 2a, asking participants to rate the same forecasters for agreement and changed opinion but with between‐subjects and within‐subjects comparisons interchanged. All comparisons that in 2a were between subjects were therefore in 2b performed by the same participants; and within‐ subjects ratings in 2a were in 2b made by participants in different conditions. Most importantly, expert pairs using same directional terms were compared between subjects to expert pairs using opposite directional terms in Experiment 2b. 5.1 | 5.1.1 Method | Participants | Changed opinions As can be seen from the lower part of Table 3, the results from Experiment 2a were replicated. Experts who change their statement from over to under (Condition A) are regarded to have changed their opinions more than experts who change their numerical probabilities without changing directional term (Condition B). The within differences are not significant. Observe also that over–under comparisons (Condition A) lead to larger variances than over–over or under– under comparisons (Condition B), in both tables. This makes sense, as over/under differences can, in principle, range from small to very large. 6 | EXPERIMENT 3 In Experiments 1a and 1b, the effect of framing on agreement was observed for statements that differed by 10 percentage points Participants were 231 students at a Norwegian university (166 regarding the focal outcome, whereas in Experiments 2a and 2b, women and 46 men, 19 did not indicate gender; median age 20 years) the effect of directional term was found when the probability esti- answering questionnaires in a break between lectures. They were ran- mates of experts using the same term differed by 20 percentage domly allocated to two different conditions. Within both conditions, points. However, with larger differences between numerical proba- the orders of statements were counterbalanced. bilities, one might expect that probabilities would become more important than the way they are framed, reducing or even reversing 5.1.2 | Materials and procedure the observed effects. Thus, Experiment 3 investigated whether the effects of framing and directional terms on perceived agreement Material and procedure were identical to those in Experiment 2a, would be robust under variations of the numerical probabilities except that participants in Condition A rated expert pairs that only dif- involved. fered in directionality (experts in one pair estimated chances to be over or under 50%, whereas the other pair disagreed about over or 6.1 | Method under 70% chance). Condition B received two expert pairs that disagreed about the numerical probabilities but used either lower or 6.1.1 | Participants upper bounds (same directionality for the experts in each pair). The second scenario asked whether the expert had changed her or his U.S. residents were recruited on MTurk and received a payment of mind when changing directionality (Condition A) or probability esti- $0.30 for completing the questionnaire. After exclusion of respon- mate (Condition B). See Tables 2 and 3 for an overview of all ratings dents who failed a simple attention check or who spent less than in both experiments. 60 s on the questionnaire, there were 242 participants, 86 women and 156 men, with a median age of 32 years; and 85.1% reported 5.2 | Results to have at least some college education. Participants were randomly allocated to receive one out of six versions of the framing 5.2.1 | Agreement between experts scenario, and one out of five versions of the directional terms scenario, as described below. The order of the two scenarios was Agreement was higher in Condition B, where both experts suggested counterbalanced. probabilities over or both suggested probabilities under (see lower panel of Table 2), even if the probabilities of the experts deviated by 6.2 | Materials and procedure 20 percentage points, as opposed to Condition A, where they indicated probabilities higher and lower than the same boundary estimate. 6.2.1 | Framing scenario This replicated the within‐subjects results of Experiment 2a. As in the previous experiment, over/under 50% and over/under 70% were The participants read a brief vignette about two biologists giving their found to indicate similar levels of disagreement, whereas experts predictions about the (fictional) critically endangered Silverback tree LØHRE ET AL. frog. In all conditions, the first biologist stated “There is an 85% prob- 9 6.3 | Results ability that the Silverback tree frog will become extinct”. The statement of the second biologist varied according to the difference in 6.3.1 | Framing probability level (5, 25, or 45 percentage points lower than those of Biologist 1) and frame (same vs. opposite frame), in a 3 × 2 Figure 1 displays the agreement ratings in the different conditions and between‐subjects design. To illustrate, when the probability level of shows that perceived agreement was generally higher when the biolo- the second biologist was 45 percentage points lower, the statement gists' statements were framed in the same way (M = 4.68, SD = 1.46) was “There is a 40% probability that the Silverback tree frog will rather than in opposite ways (M = 3.91, SD = 1.88). A 3 × 2 ANOVA become extinct” in the same frame condition and “There is a 60% with difference in probability level (5, 25, or 45 percentage points probability that the Silverback tree frog will not become extinct” in lower than those of Biologist 1) and frame (same vs. opposite) as fac- the opposite‐frame condition. After reading the statements, partici- tors confirmed a main effect of framing, F (1, 236) = 17.635, p < .001, pants rated to what extent the biologists seemed to agree with each with a medium effect size, Cohen's d = 0.46. The ANOVA also showed other on a scale from 1 (disagree completely) to 7 (agree completely). a main effect of the difference in probability level, F (2, 236) = 39.850, Our hypothesis was that for statements within the same frame, there p < .001, η2p = .252, and a significant interaction between the two fac- would be a linear decrease in perceived agreement with increasing dif- tors, F (2, 236) = 4.511, p = .012, η2p = .037. ference between the first and second biologists' probability estimates. To follow up the interaction, we compared the means for the We expected agreement ratings to be lower overall for statements in same‐ and opposite‐frame conditions for each of the three differences the opposite frame, and that the numerical difference (with regard to in probability level. When the difference between probability esti- the original outcome, i.e., extinction) would matter less for opposite‐ mates was small, agreement ratings were not significantly higher in framed statements. the same‐frame condition than in the opposite‐frame condition, Mdiff = 0.48, t(75.036) = 1.379, p = .172; and the same pattern occurred when the difference between probability estimates was 6.2.2 | Directional term scenario large, Mdiff = 0.30, t(68.449) = 1.077, p = .285. However, when there was a medium difference between probability estimates, the agree- The second scenario concerned the (fictional) Blue Rock glacier, with ment ratings were clearly higher in the same‐frame condition, two pairs of geologists giving their opinions of whether the glacier Mdiff = 1.57, t(70.326) = 4.790, p < .001. Hence, although the ratings would melt completely by the year 2050. In all conditions, the first overall were higher in the same‐frame condition than in the pair of experts stated that it was “more than 80% likely” (Geologist opposite‐frame condition, this difference was most conspicuous when A) versus “less than 80% likely” (Geologist B) that the glacier would there was a medium‐sized (25 percentage points) difference between melt. Hence, they gave the same numerical probability, but with the probability estimates involved. opposite directional terms. The second pair of experts used the same directional term, but with different numerical probability estimates. The distance between the numerical estimates of the second 6.3.2 | Directional terms pair of experts was 20, 30, 40, 50, or 60 percentage points, in five different between‐subjects conditions. Thus, Geologist C stated that As shown in Figure 2, the mean agreement ratings for the opposite‐ it was “more than 60% (50%/40%/30%/20%) likely,” whereas Geol- direction pair (which was identical in all conditions, “more than 80%” ogist D said “more than 80% likely” that the glacier would melt in all vs. “less than 80%”) all lay below the midpoint of the scale (4.0). Agree- conditions. In other words, we here employed a mixed 2 × 5 design, ment ratings for the same‐direction pair decreased according to the with direction (opposite vs. same) varied within subjects, and numer- distance between the probability estimates. A 2 × 5 mixed ANOVA ical distance between the estimates of the second (same‐term) pair with direction (same vs. opposite) as within‐factor and numerical dis- varied between subjects. The participants were asked to rate to tance as between‐factor showed a clear main effect of direction, what extent each pair of experts seemed to agree with each other F (1, 237) = 24.637, p < .001, Cohen's d = 0.29, due to the ratings on a scale from 1 (disagree completely) to 7 (agree completely). We for the same‐direction pair overall being higher (M = 4.07, SD = 1.53) hypothesized that the pair using same directional terms would than for the opposite‐direction pair (M = 3.36, SD = 1.90). The analysis receive higher ratings of agreement than the pair using opposite also showed a main effect of numerical distance, F (4, 237) = 6.060, directional terms, until the numerical distance became quite large. p < .001, η2p = .093, and an interaction between the two factors, For instance, when one expert gives a likelihood of “more than F (4, 237) = 7.749, p < .001, η2p = .116. 30%” and another expert “more than 80%,” they might not be seen To investigate the interaction, we performed separate paired‐ as being very high in agreement, even if they use the same sample t tests for each of the numerical distances. The same‐direction directional term. With such a large numerical difference, one could pair was rated significantly higher than the opposite‐direction pair even argue that the pair using opposite directional terms for when the distance between the estimates of the same‐direction pair the same number (“less than 80” vs. “more than 80”) would be in was 20, 30, and 40 percentage points (all ps < .001). Only when the better agreement. numerical distance was 50 or 60 percentage points did the same‐ LØHRE 10 ET AL. FIGURE 1 Mean ratings of agreement (1–7, higher numbers indicate more agreement) of statements from two experts, depending on the numerical difference between their probability estimates and on whether the statements were given in the same frame or in opposite frames, in Experiment 3. Error bars: ±1 SEM FIGURE 2 Mean ratings of agreement (1–7, higher numbers indicate more agreement) for statements from two pairs of experts, depending on whether they used the same or opposite directional terms, and on the numerical difference between the probability estimates of the same‐term pair of experts, in Experiment 3. Error bars: ±1 SEM direction pair receive lower agreement ratings than the opposite‐ direction pair, but still not significantly so (ps > .32). Similarly, two statements featuring the same directional term (“more than”) were seen to agree more than two statements featuring opposite directional terms (“more than” vs. “less than”). This was the 6.4 | Discussion The results of Experiment 3 confirmed the findings of the previous experiments: Framing and directional terms influence perceived (dis)agreement. When two probability statements are given in the same frame, they are seen to agree more than logically equivalent case even when the numerical difference between statements using the same term was as large as 40% (“more than 40%” vs. “more than 80%”); only for larger differences than this was the perceived level of agreement similar to two statements with the same number, but opposite directional terms (“more than 80%” vs. “less than 80%”). This shows that the effect of directional terms is remarkably robust. statements given in opposite frames. Note that although the overall effect of framing was clear, it was most pronounced (and statistically 7 | EXPERIMENT 4 significant) when the distance between the numerical probabilities involved was medium rather than large or small. Thus, there are While the previous experiments used agreement as the dependent boundaries for the effect of framing on perceived agreement, or at variable, this final experiment used agreement as an independent var- least the effect may not be equally strong for all possible combina- iable. Participants received a probability statement from one expert tions of numerical probabilities. and were asked whether it would be natural for another expert to LØHRE ET AL. 11 use a statement with the same or with a different frame, depending on depending on whether she expressed agreement or disagreement, and whether she or he expressed agreement or disagreement with the first on whether she was more or less sure than the original expert. expert. In light of the results reported so far, we hypothesized that it On the next page, participants were informed which statements would be more natural to express agreement by using statements in the experts had actually made. Camilla Dahl was said to have used the same frame, whereas disagreement would more naturally be the same frame as Anne Berg in the conditions where she agreed expressed by statements with opposing frames. The experiment also with her, and the opposite frame when she disagreed. Participants included a test of numeracy. Some studies have shown that less were asked to indicate which expert seemed to have the most numerate participants display stronger framing effects (Peters et al., expertise (four‐point scale: Anne Berg seems to have much more/a lit- 2006), whereas other studies show that numeracy is not related to tle bit more expertise, Camilla Dahl seems to have a little bit more/much susceptibility to framing (Mandel & Kapler, 2018). Hence, we wanted more expertise) and to indicate how likely they thought it was that to explore whether people's understanding of statistics relates to their the solar panels would cover the factory's energy demands (four‐ thinking about expert disagreement in the context of framing. point scale: It is very/quite likely the solar panels will cover the energy demands, It is quite/very likely the solar panels will not cover the energy 7.1 7.1.1 | demands). These two measures were included for exploratory Method | purposes. After answering some unrelated questions, participants were given Participants a numeracy test with seven items taken from Cokely, Galesic, Schulz, The participants were students at the University of Oslo who completed the questionnaire in exchange for course credit. There were Ghazal, and Garcia‐Retamero (2012) and Schwartz, Woloshin, Black, and Welch (1997). 180 participants (72.8% female), with a mean age of 23.2 years (SD = 4.9). Thirteen of these participants did not respond to the numeracy questions given in the end of the questionnaire but were 7.2 | Results 6 still included in the main analysis. 7.2.1 7.1.2 | Materials and procedure | Naturalness ratings As is clear from Table 4, participants generally found statements given in the same frame as the first expert to be more natural (M = 3.51, The scenario described prognoses about solar power production. Par- SD = 1.10) than statements given in a different frame (M = 2.33, ticipants were given statements from two experts concerning whether SD = 1.09). This was confirmed by a mixed 2 × 2 × 2 ANOVA with the solar panels being installed on the factory roof of a company in frame as within‐subjects factor and agreement (agree vs. disagree) Western Norway would produce enough electricity to cover the and probability (higher vs. lower) as between‐subjects factors, which power demands of the factory. The statements were varied between showed a large main effect of frame, F (1, 176) = 92.021, p < .001, subjects in a 2 × 2 design, with agreement/disagreement as the first η2p = .343. factor and target probability (the second expert giving a probability Importantly, the analysis also showed an interaction between 10 percentage points higher vs. 10 percentage points lower than the frame and agreement, F (1, 176) = 19.091, p < . 001, η2p = .098. first expert) as the second factor (Table 4 gives an overview of the Table 4 indicates that this interaction is due to statements in the same statements in the different conditions). Note that the difference of frame being seen as more natural to express agreement (M = 3.75, 10 percentage points is similar to that used in Experiments 1a and SD = 1.09) than to express disagreement (M = 3.27, SD = 1.05), 1b. In all conditions, the first expert, Anne Berg, gave the following whereas statements in the opposite frame were rated as less suited statement to a local newspaper: “My judgment is that there is a 55% to express agreement (M = 2.03, SD = 0.95) than to express disagree- probability that the solar panels will cover the power demands of ment (M = 2.63, SD = 1.15). Simple effects analysis confirmed that the factory”. A second expert, Camilla Dahl, said: “I quite agree with these differences were significant for statements in the same frame, Anne Berg …” (agreement condition) or “I quite disagree with Anne F (1, 176) = 9.607, p = .002, Cohen's d = 0.45, as well as for statements Berg …” (disagreement condition), with two potential continuations, in the opposite frame, F (1, 176) = 13.936, p < .001, Cohen's one stating a 45% (65%) probability that the solar panels would cover d = −0.57. Frame did not interact with probability, nor was there a the power demands of the factory and another including the comple- three‐way interaction, F s < 1.9. mentary probability, 55% (35%), that the solar panels would not cover Although there were no main effects of either agreement or prob- the power demands of the factory. The participants were asked to rate ability, there was a significant interaction between these two how natural they found it for the second expert to use a statement of between‐subjects factors, F (1, 176) = 7.924, p = .005, η2p = .043. This the same or the opposite frame as the original expert (using a scale effect appears to be due to higher probabilities in general being seen from 1 [does not sound natural at all] to 5 [sounds completely natural]), as more natural for agreement than for disagreement, whereas lower 6 The results were similar if only participants who completed all parts of the questionnaire were included. probabilities were thought to be more natural as expressions of disagreement than of agreement. LØHRE 12 ET AL. TABLE 4 Mean naturalness ratings (1–5, higher numbers indicate higher naturalness) of probability statements from a second expert, depending on claimed agreement or disagreement with the first expert who gave a 55% probability, in Experiment 4 Frame Agree Disagree Higher (X = 65%, Y = 35%) n = 43 Lower (X = 45%, Y = 55%) n = 46 Higher (X = 65%, Y = 35%) n = 45 Lower (X = 45%, Y = 55%) n = 46 Same frame (X% probability that the solar panels will cover the power demands) 3.95 (1.02) 3.57 (1.13) 3.02 (1.08) 3.52 (0.98) Opposite frame (Y% probability that the solar panels will not cover the power demands) 2.16 (0.97) 1.91 (0.91) 2.64 (1.19) 2.61 (1.13) 7.2.2 Participants in Experiment 4 also found agreement statements Ratings of expertise and likelihood | more natural with higher probabilities and disagreement statements The expertise ratings did not vary between conditions, F s < 1, but more natural with lower probabilities. This finding can be understood showed that participants had a slight preference for Anne Berg (the in light of the logic of the number system, where higher numbers first expert), with 62.7% of participants stating that she seemed to entail lower ones, but not vice versa. To exemplify, when one expert have much or a little bit more expertise. The ratings of how likely it says the probability is 55%, a second expert could say “Yes, it is was that the solar panels would cover the factory's energy demands 55%, it is even 65%”. However, a lower probability, like 45%, does were influenced by the second expert's probability level, F (1, not contain the original number and may therefore be seen to express 175) = 6.337, p = .013, η2p = .035. When Camilla Dahl gave a higher disagreement to a greater extent (see also Løhre, 2018; Experiment 2). probability than Anne Berg, 69.0% of the participants thought it was This finding implies that two experts may be seen as being more or very or quite likely the panels would cover the energy demands, but less in agreement depending on which expert expresses their opinion when she gave a lower probability, a slight majority (53.3%) thought first. it was very or quite likely the panels would not cover the energy demands. 8 7.2.3 | | GENERAL DISCUSSION The role of numeracy The persuasive power of expert agreement has been evidenced in a The 167 participants who completed the numeracy test on average number of studies (e.g., Aklin & Urpelainen, 2014; Lewandowsky had a numeracy score of 4.23 (SD = 1.65), with no significant differ- et al., 2013; Nagler, 2014; van der Linden et al., 2015). With this in ences between conditions, F s < 2.3. Numeracy did not significantly mind, it is important to gain knowledge about factors that influence correlate with the naturalness ratings for statements in the same whether lay people perceive experts to agree or to disagree. The cur- (r = .028, p = .717) or different frame (r = −.133, p = .086). Neither rent investigation demonstrates that for expert opinions expressed as were there any significant correlations when each condition was numerical probability estimates, choices made in the communication inspected separately, all ps > .065. Thus, numeracy did not (at least process can be consequential for perceived agreement. In Experiment in this design) have any clear influence on perceptions of 1a, two experts using numerical probability estimates focusing on dif- disagreement. ferent, complementary outcomes, that is, differently framed statements, were perceived as disagreeing more than two experts using the same frame. Experiment 1b extended this finding to within‐expert 7.3 | Discussion agreement: An expert who changed the framing of a probability statement was perceived as less consistent than an expert who kept the Experiment 4 showed that the way a probability statement is framed frame constant. Experiments 2a and 2b demonstrated a similar point can make it suitable for expressing agreement or disagreement with for the use of directional terms like “over” and “under”. Statements another expert. When the first expert gives the probability that an that differed in direction (e.g., “over 50%” vs. “under 50%”) were per- event or outcome will occur, a second expert can express agreement ceived as indicating greater disagreement between experts and a by also focusing on the probability that the event will occur, even larger change of opinion from one expert than statements with the for probabilities that differ from the original estimate. However, if same direction but different numerical probabilities (e.g., “over 50%” the expert wishes to express disagreement, focusing on the probability vs. “over 70%”). Experiment 3 investigated the effects of framing that the event will not occur may be seen as more natural. Note that and directional terms for a larger range of numerical differences. The people in general find it more natural to talk about the probability that effect of frames was most pronounced when the numerical difference an event will occur, but even so, probabilities focusing on nonoccur- between estimates was medium (25 percentage points) rather than rence are acceptable to express disagreement. large (45 percentage points) or small (five percentage points), although LØHRE ET AL. 13 overall same‐frame statements were seen as more agreeing than probabilities that converge towards an intermediate number (e.g., over opposite‐frame statements. The effect of directional term was stable 50% vs. under 70%). This is an interesting empirical question that for numerical differences up to 40 percentage points, with same direc- could be investigated in future research. A second limitation is that tional term indicating higher agreement than opposing directional all scenarios were restricted to an environmental domain. However, term. Finally, participants in Experiment 4 found it more natural for as framing effects are demonstrated in a wide range of domains an expert who is expressing agreement with another expert to give a including finance (Teigen & Nikolaisen, 2009), health (Sanford et al., numerical probability within the same frame, whereas a probability 2002), and sports (Leong, McKenzie, Sher, & Müller‐Trede, 2017; statement in the opposite frame was rated as more natural when used Yeung, 2014), we expect that our findings will generalize to other to express disagreement. domains. Theoretically, our results are in line with both the information leak- This investigation also provided some evidence that using framing age perspective on framing (Sher & McKenzie, 2006) and with fuzzy‐ to manipulate agreement can have downstream consequences. In trace theory (Reyna, 2004). A listener receiving two probability esti- Experiment 1a, both the trust in the experts and in research and the mates expressed in different frames might infer that the speakers have perceived risk of air pollution were lowest in Condition B, where the different perspectives on the outcome in question: A speaker who two experts gave statements in opposite frames. A similar (although gives a 45% probability that the air pollution will not lead to negative not statistically significant) pattern of results was observed in Experi- health effects presumably holds the opinion that the problem of air ment 1b, where the statements were attributed to one expert at two pollution is not that serious, whereas experts giving a 45% or 55% points in time. This finding should be interpreted with caution, but probability that the air pollution will lead to negative health effects given the existing research on the effects of expert agreement on probably agree that air pollution may have negative consequences. lay people's attitudes and behavior, it is not unrealistic to expect that In other words, different frames contain qualitatively different gist the choice of same versus different frame could influence trust and messages. The results regarding the directional terms “over” and risk perception. “under” can be interpreted in a similar way. When used in combination The present investigation does not address how robust the effects with numerical probabilities, these terms (like frames) serve to give the of framing and directional terms are when combined with other infor- listener implicit information about the speaker's opinions or degree of mation. A recent handbook for uncertainty communication (Corner, concern: “over” points towards the possible occurrence, whereas Lewandowsky, Phillips, & Roberts, 2015) recommends that communi- “under” is a signal that the outcome actually might not occur (Hohle cators underscore areas where agreement is high and the science is & Teigen, 2018a). effectively settled, for example, the fact that greenhouse gases are a The present results show that frames and directional terms shaped leading cause of climate change. It might be that if people are first perceptions of expert agreement both between different experts and informed of this general consensus, they would be willing to accept for one expert over time, and across different samples (students and more disagreement about specific topics. Relatedly, if people are made MTurk participants), languages (Polish, Norwegian, and English), set- to understand that science is an ongoing process and an arena of tings (online and paper‐pencil), and designs (within vs. between sub- debate, not a fixed body of facts, they are more persuaded by mes- jects). Nevertheless, there are at least two limitations regarding the sages communicating high uncertainty (Rabinovich & Morton, 2012). generalizability of our results. First, the difference in probability level This points to several possible moderators for perceived disagreement, between experts' statements may be a boundary condition for the including general views of science and scientists, political views and effects. Although statements in the same frame were generally rated ideology, and specific knowledge about topic areas (see Dieckmann as agreeing more than statements in opposite frames, the effect was et al., 2017; Johnson & Dieckmann, 2017). significant only for medium (10 and 25 percentage points difference) In the current article, participants were exposed either to two dif- values, and not for large and small differences. The effect for direc- ferent experts or to two statements from the same expert. However, tional terms was robust below 40 percentage points of difference people may also be exposed to revised opinions from two or more between statements. However, other combinations of probabilities experts. One recent study demonstrated that when people receive and directional terms could be informative. The present article only revised probability estimates from two experts, they find these two compared different directional terms that diverged from the same experts to agree more if they have revised their estimate in the same numerical probability (e.g., over 50% vs. under 50%). The magnitude rather than in opposite directions (Hohle & Teigen, 2018b). This adds of this difference is indeterminate, as “over 50%” and “under 50%” to previous findings of a trend effect in revisions of probability esti- could, in principle, mean a whole range of probabilities. However, we mates (Erlandsson, Hohle, Løhre, & Västfjell, 2018; Hohle & Teigen, know from previous research (Ferson et al., 2015; Hohle & Teigen, 2015; Maglio & Polman, 2016), where people expect that revisions 2018a, see also Footnote 5; Teigen et al., 2007) that such expressions will continue in the same direction (e.g., a probability that has are usually taken to indicate amounts that are rather close to the ref- increased from 60% to 70% will become even higher in the future). erence value, so the difference between an “over” and “under” esti- Given that lay people (especially those with low numeracy scores) mate is probably small rather than large. However, we do not know may experience difficulties in processing information about probability the exact magnitude of this difference, or how people will judge a (Cokely et al., 2012; Traczyk & Fulawka, 2016) and rarely search for comparison such information (Huber, 2012; Traczyk et al., 2018; Tyszka & of different directional terms involving different LØHRE 14 Zaleskiewicz, 2006), future studies should extend the work on perceived agreement to statements that do not contain numerical probabilities. It is common for experts in fields as different as climate science and intelligence analysis to use verbal probabilities (e.g., likely, possible, and uncertain) to communicate risks and likelihoods (Ho, Budescu, Dhami, & Mandel, 2015). One can hypothesize in light of the current findings that verbal probabilities with different directionality (e.g., “likely” vs. “not certain”) will be perceived as disagreeing more than verbal probabilities with the same directionality (e.g., “likely” vs. “highly likely”; Teigen & Brun, 1999). In other cases, experts may not refer to probabilities at all but could communicate risks in ways that induce more or less vivid affect‐laden imagery (Sobkow, Traczyk, & Zaleskiewicz, 2016; Traczyk, Sobkow, & Zaleskiewicz, 2015). Experts who induce different emotions in the listener may be perceived as disagreeing more than two experts who induce the same emotion but with different intensity. The current findings have practical relevance for experts communicating complex scientific topics, be it climate change, environmental risk, or personal health. In essence, the findings show that simple communication choices influence the perceived degree of expert agreement. Experts who want to give accurate information, but to avoid undermining lay people's sense of agreement among experts, should take care which frame they choose or which directional term (if any) they express. To emphasize agreement between different estimates, communicators should use corresponding framings. To accentuate disagreements, or a change of opinion, opposite framings should be used (e.g., “the chance has increased from under 40% to more than 50%”). We believe this is an important and understudied topic and look forward to future research identifying other factors that influence perceived expert agreement. ACKNOWLEDGEMEN TS We would like to thank Adrian Matukiewicz for help with data collection and Angelika Olszewska for help with data analysis for Experiments 1a and 1b. This research was supported by the Research Council of Norway (Norges Forskningsråd), Grant 235585/E10, to Erik Løhre, Sigrid Møyner Hohle, and Karl Halvor Teigen and by SWPS University of Social Sciences and Humanities (BST/Wroc/2017/A/ 08) to Agata Sobkow. CONF LICT OF INT E RE ST The authors report no conflicts of interest. ORCID Erik Løhre https://orcid.org/0000-0003-3173-013X Agata Sobkow https://orcid.org/0000-0002-5357-744X Sigrid Møyner Hohle https://orcid.org/0000-0002-4533-0500 ET AL. Aklin, M., & Urpelainen, J. (2014). Perceptions of scientific dissent undermine public support for environmental policy. Environmental Science & Policy, 38, 173–177. https://doi.org/10.1016/j.envsci.2013.10.006 Cokely, E. T., Galesic, M., Schulz, E., Ghazal, S., & Garcia‐Retamero, R. (2012). Measuring risk literacy: The Berlin Numeracy Test. Judgment and Decision Making, 7(1), 25–47. Corner, A., Lewandowsky, S., Phillips, M., & Roberts, O. (2015). The uncertainty handbook. Bristol: University of Bristol. Croco, S. E. (2016). The flipside of flip‐flopping: Leader inconsistency, citizen preferences, and the war in Iraq. Foreign Policy Analysis, 12(3), 237–257. https://doi.org/10.1093/fpa/orw006 Dieckmann, N. F., Johnson, B. B., Gregory, R., Mayorga, M., Han, P. K. J., & Slovic, P. (2017). Public perceptions of expert disagreement: Bias and incompetence or a complex and random world? Public Understanding of Science, 26(3), 325–338. https://doi.org/10.1177/ 0963662515603271 Entman, R. M. (1993). Framing: Toward clarification of a fractured paradigm. Journal of Communication, 43(4), 51–58. https://doi.org/ 10.1111/J.1460‐2466.1993.Tb01304.X Erlandsson, A., Hohle, S. M., Løhre, E., & Västfjell, D. (2018). The rise and fall of scary numbers: The effect of perceived trends on future estimates, severity ratings, and help‐allocations in a cancer context. Journal of Applied Social Psychology, 48(11), 618–633. https://doi.org/ 10.1111/jasp.12552 Falk, A., & Zimmermann, F. (2017). Consistency as a signal of skills. Management Science, 63(7), 2197–2210. https://doi.org/10.1287/ mnsc.2016.2459 Ferson, S., O'Rawe, J., Antonenko, A., Siegrist, J., Mickley, J., Luhmann, C. C., … Finkel, A. M. (2015). Natural language of uncertainty: Numeric hedge words. International Journal of Approximate Reasoning, 57, 19–39. https://doi.org/10.1016/j.ijar.2014.11.003 Ho, E. H., Budescu, D. V., Dhami, M. K., & Mandel, D. R. (2015). Improving the communication of uncertainty in climate science and intelligence analysis. Behavioral Science & Policy, 1(2), 43–55. https://doi.org/ 10.1353/bsp.2015.0015 Hohle, S. M., & Teigen, K. H. (2015). Forecasting forecasts: The trend effect. Judgment and Decision Making, 10(5), 416–428. Hohle, S. M., & Teigen, K. H. (2018a). More than 50% or less than 70% chance: Pragmatic implications of single bound probability estimates. Journal of Behavioral Decision Making, 31(1), 138–150. https://doi.org/ 10.1002/bdm.2052 Hohle, S. M., & Teigen, K. H. (2018b). When probabilities change: Perceptions and implications of trends in uncertain climate forecasts. Journal of Risk Research.. https://doi.org/10.1080/13669877.2018.1459801 Howell, D. C. (2013). Statistical methods for psychology (8th, International ed.). Wadsworth: Cengage Learning. Huber, O. (2012). Risky decisions: Active risk management. Current Directions in Psychological Science, 21(1), 26–30. https://doi.org/10.1177/ 0963721411422055 Johnson, B. B., & Dieckmann, N. F. (2017). Lay Americans' views of why scientists disagree with each other. Public Understanding of Science. https://doi.org/10.1177/0963662517738408, 27, 824–835. Kahan, D. (2017). The “Gateway Belief” illusion: Reanalyzing the results of a scientific‐consensus messaging study. Journal of Science Communication, 16(5), A03. https://doi.org/10.22323/2.16050203 RE FE R ENC E S Kahneman, D. (2011). Thinking, fast and slow. London: Penguin Group. Aczel, B., Szollosi, A., & Bago, B. (2018). The effect of transparency on framing effects in within‐subject designs. Journal of Behavioral Decision Making, 31(1), 25–39. https://doi.org/10.1002/bdm.2036 Kahneman, D., & Frederick, S. (2005). A model of heuristic judgment. In K. J. Holyoak, & R. G. Morrison (Eds.), The Cambridge handbook of thinking and reasoning (pp. 267–293). New York: Cambridge University Press. LØHRE ET AL. Leong, L. M., McKenzie, C. R. M., Sher, S., & Müller‐Trede, J. (2017). The role of inference in attribute framing effects. Journal of Behavioral Decision Making, 30(5), 1147–1156. https://doi.org/10.1002/bdm.2030 Lewandowsky, S., Gignac, G. E., & Vaughan, S. (2013). The pivotal role of perceived scientific consensus in acceptance of science. Nature Climate Change, 3(4), 399–404. https://doi.org/10.1038/Nclimate1720 Løhre, E. (2018). Stronger, sooner, and more certain climate change: A link between certainty and outcome strength in revised forecasts. Quarterly Journal of Experimental Psychology, 71(12), 2531–2547. https://doi.org/ 10.1177/1747021817746062 Maglio, S. J., & Polman, E. (2016). Revising probability estimates: Why increasing likelihood means increasing impact. Journal of Personality and Social Psychology, 111(2), 141–158. https://doi.org/10.1037/ pspa0000058 Mandel, D. R., & Kapler, I. V. (2018). Cognitive style and frame susceptibility in decision‐making. Frontiers in Psychology, 9(1461). https://doi.org/ 10.3389/fpsyg.2018.01461 McKenzie, C. R. M., & Nelson, J. D. (2003). What a speaker's choice of frame reveals: Reference points, frame selection, and framing effects. Psychonomic Bulletin & Review, 10(3), 596–602. https://doi.org/ 10.3758/Bf03196520 Mussweiler, T. (2003). Comparison processes in social judgment: Mechanisms and consequences. Psychological Review, 110(3), 472–489. https://doi.org/10.1037/0033‐295X.110.3.472 Nagler, R. H. (2014). Adverse outcomes associated with media exposure to contradictory nutrition messages. Journal of Health Communication, 19(1), 24–40. https://doi.org/10.1080/10810730.2013.798384 Oreskes, N., & Conway, E. M. (2010). Merchants of doubt: How a handful of scientists obscured the truth on issues from tobacco smoke to global warming. New York: Bloomsbury Press. Peters, E., Västfjäll, D., Slovic, P. l., Mertz, C. K., Mazzocco, K., & Dickert, S. (2006). Numeracy and decision making. Psychological science, 17(5), 407–413. https://doi.org/10.1111/j.1467‐9280.2006.01720.x 15 Sobkow, A., Traczyk, J., & Zaleskiewicz, T. (2016). The affective bases of risk perception: Negative feelings and stress mediate the relationship between mental imagery and risk perception. Frontiers in Psychology, 7. https://doi.org/10.3389/fpsyg.2016.00932 Tedeschi, J. T., Schlenker, B. R., & Bonoma, T. V. (1971). Cognitive dissonance: Private ratiocination or public spectacle. American Psychologist, 26(8), 685. https://doi.org/10.1037/h0032110 Teigen, K. H. (2008). More than X is a lot: Pragmatic implicatures of one‐ sided uncertainty intervals. Social Cognition, 26(4), 379–400. https:// doi.org/10.1521/soco.2008.26.4.379 Teigen, K. H. (2016). Framing of numerical quantities. In G. Keren, & G. Wu (Eds.), The Wiley Blackwell handbook of judgment and decision making (pp. 568–589). John Wiley & Sons, Ltd. Teigen, K. H., & Brun, W. (1999). The directionality of verbal probability expressions: Effects on decisions, predictions, and probabilistic reasoning. Organizational Behavior and Human Decision Processes, 80(2), 155–190. https://doi.org/10.1006/Obhd.1999.2857 Teigen, K. H., Halberg, A. M., & Fostervold, K. I. (2007). Single‐limit interval estimates as reference points. Applied Cognitive Psychology, 21(3), 383–406. https://doi.org/10.1002/Acp.1283 Teigen, K. H., & Nikolaisen, M. I. (2009). Incorrect estimates and false reports: How framing modifies truth. Thinking & Reasoning, 15(3), 268–293. https://doi.org/10.1080/13546780903020999 Traczyk, J., & Fulawka, K. (2016). Numeracy moderates the influence of task‐irrelevant affect on probability weighting. Cognition, 151, 37–41. https://doi.org/10.1016/j.cognition.2016.03.002 Traczyk, J., Lenda, D., Serek, J., Fulawka, K., Tomczak, P., Strizyk, K., … Sobkow, A. (2018). Does fear increase search effort in more numerate people? An experimental study investigating information acquisition in a decision from experience task. Frontiers in Psychology, 9, 1203. https://doi.org/10.3389/fpsyg.2018.01203 Traczyk, J., Sobkow, A., & Zaleskiewicz, T. (2015). Affect‐laden imagery and risk taking: The mediating role of stress and risk perception. Plos One, 10(3). https://doi.org/10.1371/journal.pone.0122226 Rabinovich, A., & Morton, T. A. (2012). Unquestioned answers or unanswered questions: Beliefs about science guide responses to uncertainty in climate change risk communication. Risk Analysis, 32(6), 992–1002. https://doi.org/10.1111/j.1539‐6924.2012.01771.x Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 211(4481), 453–458. https://doi.org/ 10.1126/Science.7455683 Reyna, V. F. (2004). How people make decisions that involve risk—A dual‐ processes approach. Current Directions in Psychological Science, 13(2), 60–66. https://doi.org/10.1111/j.0963‐7214.2004.00275.x Tversky, A., & Kahneman, D. (1986). Rational choice and the framing of decisions. Journal of Business, 59(4), S251–S278. https://doi.org/ 10.1086/296365 Reyna, V. F., & Brainerd, C. J. (1991). Fuzzy‐trace theory and framing effects in choice: Gist extraction, truncation, and conversion. Journal of Behavioral Decision Making, 4(4), 249–262. https://doi.org/ 10.1002/bdm.3960040403 Tyszka, T., & Zaleskiewicz, T. (2006). When does information about probability count in choices under risk? Risk Analysis, 26(6), 1623–1636. https://doi.org/10.1111/j.1539‐6924.2006.00847.x Sanford, A. J., Fay, N., Stewart, A., & Moxey, L. (2002). Perspective in statements of quantity, with implications for consumer psychology. Psychological science, 13(2), 130–134. https://doi.org/10.1111/1467‐ 9280.00424 Schwartz, L. M., Woloshin, S., Black, W. C., & Welch, H. G. (1997). The role of numeracy in understanding the benefit of screening mammography. Annals of Internal Medicine, 127(11), 966–972. Sher, S., & McKenzie, C. R. M. (2006). Information leakage from logically equivalent frames. Cognition, 101(3), 467–494. https://doi.org/ 10.1016/J.Cognition.2005.11.001 Sher, S., & McKenzie, C. R. M. (2008). Framing effects and rationality. In N. Chater, & M. Oaksford (Eds.), The probabilistic mind: Prospects for Bayesian cognitive science (pp. 79–96). Oxford: Oxford University Press. Smithson, M. (1999). Conflict aversion: Preference for ambiguity vs conflict in sources and evidence. Organizational Behavior and Human Decision Processes, 79(3), 179–198. https://doi.org/10.1006/obhd.1999.2844 van der Linden, S. L., Leiserowitz, A. A., Feinberg, G. D., & Maibach, E. W. (2015). The scientific consensus on climate change as a gateway belief: Experimental evidence. Plos One, 10(2). https://doi.org/10.1371/journal.pone.0118489 van der Linden, S. L., Leiserowitz, A. A., & Maibach, E. W. (2017). Gateway illusion or cultural cognition confusion. Journal of Science Communication, 16(5), A04. https://doi.org/10.17863/CAM.22574 Yeung, S. (2014). Framing effect in evaluation of others' predictions. Judgment and Decision Making, 9(5), 445–464. How to cite this article: Løhre E, Sobkow A, Hohle SM, Teigen KH. Framing experts' (dis)agreements about uncertain environmental events. J Behav Dec Making. 2019;1–15. https://doi.org/10.1002/bdm.2132