Barbara Mellers
University of Pennsylvania, The Wharton School, Faculty Member
Research Interests:
Research Interests:
Research Interests:
Research Interests:
Research Interests:
Research Interests:
Research Interests:
Research Interests:
Research Interests:
Research Interests:
In dynamic task environments, decision makers are vulnerable to two types of errors: sticking too closely to the rules (excessive conformity) or straying too far from them (excessive deviation). We explore the effects of process and... more
In dynamic task environments, decision makers are vulnerable to two types of errors: sticking too closely to the rules (excessive conformity) or straying too far from them (excessive deviation). We explore the effects of process and outcome accountability on the susceptibility to these errors. Using a multiple‐cue probability‐learning task, we show that process accountability encourages conformity errors and outcome accountability promotes deviation errors. Two additional studies explore the moderating effects of self‐focused and other‐focused group norms. Self‐focused norms reduce the effect of process accountability on excessive conformity. Other‐focused norms reduce the effect of outcome accountability on excessive deviation. Our results qualify prevailing claims about the benefits of process over outcome accountability and show that those benefits hinge on prevailing group norms, on the effectiveness of prescribed decision rules, and on the amount of irreducible uncertainty in the prediction task. Copyright © 2016 John Wiley & Sons, Ltd.
Research Interests:
Research Interests:
Research Interests:
From 2011 to 2015, the U.S. intelligence community sponsored a series of forecasting tournaments that challenged university-based researchers to invent measurably better methods of forecasting political events. Our group, the Good... more
From 2011 to 2015, the U.S. intelligence community sponsored a series of forecasting tournaments that challenged university-based researchers to invent measurably better methods of forecasting political events. Our group, the Good Judgment Project, won these tournaments by balancing the collaboration and competition of members across disciplines. At the outset, psychologists were ahead of economists in identifying individual differences in forecasting skill and developing methods of debiasing forecasts, whereas economists were ahead of psychologists in designing simple market mechanisms that distilled predictive signals from noisy individual-level data. Working closely with statisticians, psychologists eventually beat the markets by producing better probability estimates that funneled top forecasters into elite teams and aggregated their judgments using a log-odds formula tuned to the diversity of the forecasters. Our research group performed best when team members strove to get as much as possible from their home disciplines, but acknowledged their limitations and welcomed help from outsiders. (PsycINFO Database Record (c) 2019 APA, all rights reserved).
Research Interests:
Research Interests:
Research Interests:
A four-year series of subjective probability forecasting tournaments sponsored by the U.S. intelligence community revealed a host of replicable drivers of predictive accuracy, including experimental interventions such as training in... more
A four-year series of subjective probability forecasting tournaments sponsored by the U.S. intelligence community revealed a host of replicable drivers of predictive accuracy, including experimental interventions such as training in probabilistic reasoning, anti‐groupthink teaming, and tracking of talent. Drawing on these data, we propose a Bayesian BIN model (Bias, Information, Noise) for disentangling the underlying processes that enable forecasters and forecasting methods to improve—either by tamping down bias and noise in judgment or by ramping up the efficient extraction of valid information from the environment. The BIN model reveals that noise reduction plays a surprisingly consistent role across all three methods of enhancing performance. We see the BIN method as useful in focusing managerial interventions on what works when and why in a wide range of domains. An R-package called BINtools implements our method and is available on the first author’s personal website. This paper was accepted by Manel Baucells, decision analysis.
Research Interests:
We use results from a multiyear, geopolitical forecasting tournament to highlight the ability of the contribution weighted model [Budescu DV, Chen E (2015) Identifying expertise to extract the wisdom of crowds. Management Sci.... more
We use results from a multiyear, geopolitical forecasting tournament to highlight the ability of the contribution weighted model [Budescu DV, Chen E (2015) Identifying expertise to extract the wisdom of crowds. Management Sci. 61(2):267–280] to capture and exploit expertise. We show that the model performs better when judges gain expertise from manipulations such as training in probabilistic reasoning and collaborative interaction from serving on teams. We document the model’s robustness using probability judgments from early, middle, and late phases of the forecasting period and by showing its strong performance in the presence of hypothetical malevolent forecasters. The model is highly cost-effective: it operates well, even with random attrition, as the number of judges shrinks and information on their past performance is reduced.
Research Interests:
Research Interests:
Research Interests:
The heuristics-and-biases research program highlights reasons for expecting people to be poor intuitive forecasters. This article tests the power of a cognitive-debiasing training module (“CHAMPS KNOW”) to improve probability judgments in... more
The heuristics-and-biases research program highlights reasons for expecting people to be poor intuitive forecasters. This article tests the power of a cognitive-debiasing training module (“CHAMPS KNOW”) to improve probability judgments in a four-year series of geopolitical forecasting tournaments sponsored by the U.S. intelligence community. Although the training lasted less than one hour, it consistently improved accuracy (Brier scores) by 6 to 11% over the control condition. Cognitive ability and practice also made largely independent contributions to predictive accuracy. Given the brevity of the training tutorials and the heterogeneity of the problems posed, the observed effects are likely to be lower-bound estimates of what could be achieved by more intensive interventions. Future work should isolate which prongs of the multipronged CHAMPS KNOW training were most effective in improving judgment on which categories of problems.
Research Interests:
Several studies have demonstrated that income inequality has risen since the 1960s. Other studies have found that people underestimate the extent of the inequality. Reasons for these mis-perceptions include over-reliance on one’s own... more
Several studies have demonstrated that income inequality has risen since the 1960s. Other studies have found that people underestimate the extent of the inequality. Reasons for these mis-perceptions include over-reliance on one’s own local environment and ideologically-motivated reasoning. We propose a novel mechanism to account for the mis-perceptions of income inequality. We posit that the degree to which people believe that they have control over their lives, their perceived autonomy, is related to: (1) the belief that inequality is low and, furthermore, (2) the belief that these inequalities are fair. Using a representative sample of 3,427 Americans, we find evidence to support these hypotheses.
Research Interests:
Research Interests:
We report the results of the first large-scale, long-term, experimental test between two crowdsourcing methods: prediction markets and prediction polls. More than 2,400 participants made forecasts on 261 events over two seasons of a... more
We report the results of the first large-scale, long-term, experimental test between two crowdsourcing methods: prediction markets and prediction polls. More than 2,400 participants made forecasts on 261 events over two seasons of a geopolitical prediction tournament. Forecasters were randomly assigned to either prediction markets (continuous double auction markets) in which they were ranked based on earnings, or prediction polls in which they submitted probability judgments, independently or in teams, and were ranked based on Brier scores. In both seasons of the tournament, prices from the prediction market were more accurate than the simple mean of forecasts from prediction polls. However, team prediction polls outperformed prediction markets when forecasts were statistically aggregated using temporal decay, differential weighting based on past performance, and recalibration. The biggest advantage of prediction polls was at the beginning of long-duration questions. Results suggest...
Research Interests:
Research Interests:
Research Interests:
Research Interests:
Research Interests:
This article proposes an Item Response Theoretical (IRT) forecasting model that incorporates proper scoring rules and provides evaluations of forecasters’ expertise in relation to the features of the specific questions they answer. We... more
This article proposes an Item Response Theoretical (IRT) forecasting model that incorporates proper scoring rules and provides evaluations of forecasters’ expertise in relation to the features of the specific questions they answer. We illustrate the model using geopolitical forecasts obtained by the Good Judgment Project (GJP) (see Mellers, Ungar, Baron, Ramos, Gurcay, Fincher, Scott, Moore, Atanasov, Swift, Murray, Stone & Tetlock, 2014). The expertise estimates from the IRT model, which take into account variation in the difficulty and discrimination power of the events, capture the underlying construct being measured and are highly correlated with the forecasters’ Brier scores. Furthermore, our expertise estimates based on the first three years of the GJP data are better predictors of both the forecasters’ fourth year Brier scores and their activity level than the overall Brier scores obtained and Merkle’s (2016) predictions, based on the same period. Lastly, we discuss the benef...
Research Interests:
Good judgment is often gauged against two gold standards – coherence and correspondence. Judgments are coherent if they demonstrate consistency with the axioms of probability theory or propositional logic. Judgments are correspondent if... more
Good judgment is often gauged against two gold standards – coherence and correspondence. Judgments are coherent if they demonstrate consistency with the axioms of probability theory or propositional logic. Judgments are correspondent if they agree with ground truth. When gold standards are unavailable, silver standards such as consistency and discrimination can be used to evaluate judgment quality. Individuals are consistent if they assign similar judgments to comparable stimuli, and they discriminate if they assign different judgments to dissimilar stimuli. We ask whether “superforecasters”, individuals with noteworthy correspondence skills (see Mellers et al., 2014) show superior performance on laboratory tasks assessing other standards of good judgment. Results showed that superforecasters either tied or out-performed less correspondent forecasters and undergraduates with no forecasting experience on tests of consistency, discrimination, and coherence. While multifaceted, good ju...
Research Interests:
Research Interests:
Five university-based research groups competed to recruit forecasters, elicit their predictions, and aggregate those predictions to assign the most accurate probabilities to events in a 2-year geopolitical forecasting tournament. Our... more
Five university-based research groups competed to recruit forecasters, elicit their predictions, and aggregate those predictions to assign the most accurate probabilities to events in a 2-year geopolitical forecasting tournament. Our group tested and found support for three psychological drivers of accuracy: training, teaming, and tracking. Probability training corrected cognitive biases, encouraged forecasters to use reference classes, and provided forecasters with heuristics, such as averaging when multiple estimates were available. Teaming allowed forecasters to share information and discuss the rationales behind their beliefs. Tracking placed the highest performers (top 2% from Year 1) in elite teams that worked together. Results showed that probability training, team collaboration, and tracking improved both calibration and resolution. Forecasting is often viewed as a statistical problem, but forecasts can be improved with behavioral interventions. Training, teaming, and tracki...
Surprise is a fundamental link between cognition and emotion. It is shaped by cognitive assessments of likelihood, intuition, and superstition, and it in turn shapes hedonic experiences. We examine this connection between cognition and... more
Surprise is a fundamental link between cognition and emotion. It is shaped by cognitive assessments of likelihood, intuition, and superstition, and it in turn shapes hedonic experiences. We examine this connection between cognition and emotion and offer an explanation called decision affect theory. Our theory predicts the affective consequences of mistaken beliefs, such as overconfidence and hindsight. It provides insight about why the pleasure of a gain can loom larger than the pain of a comparable loss. Finally, it explains cross-cultural differences in emotional reactions to surprising events. By changing the nature of the unexpected (from chance to good luck), one can alter the emotional reaction to surprising events.
Research Interests:
Although inequality in the US has increased since the 1960s, several studies show that Americans underestimate it. Reasons include overreliance on one’s local perspective and ideologically-motivated cognition. We propose a novel mechanism... more
Although inequality in the US has increased since the 1960s, several studies show that Americans underestimate it. Reasons include overreliance on one’s local perspective and ideologically-motivated cognition. We propose a novel mechanism to account for the misperceptions of income inequality. We hypothesize that compared to those who feel less autonomy, the people who believe they are autonomous and have control over their lives also believe that (1) income inequality is lower and (2) income inequality is more acceptable. Using a representative sample of 3,427 Americans, we find evidence to support these hypotheses.
Research Interests:
We introduce a new method for converting individual probability estimates (obtained through surveys) into market orders for use in a Continuous Double Auction prediction market. Our Survey-Powered Market Agent (SPMA) algorithm is based on... more
We introduce a new method for converting individual probability estimates (obtained through surveys) into market orders for use in a Continuous Double Auction prediction market. Our Survey-Powered Market Agent (SPMA) algorithm is based on actual forecaster behavior, and offers notable advantages over existing market agent algorithms such as Zero Intelligence Plus (ZIP) agents: SPMAs only require probability estimates (and not bid direction nor quantity), are more behaviorally realistic, and work well when probabilities change over time. We validate SPMA using prediction market data and probability estimates elicited through surveys from a large set of forecasters on 88 individual forecasting problems over the course of a year. SPMA outperforms simple averages of the same probability forecasts and is competitive with sophisticated opinion poll aggregation methods and prediction markets. We use a rich set of market and poll data to empirically test the assumptions behind SPMA’s operat...
Research Interests:
It is well known that goals serve as reference points, and their influence on pleasure can be understood with prospect theory’s value function. We examine how people feel about their progress on two goals (i.e., academics and fitness).... more
It is well known that goals serve as reference points, and their influence on pleasure can be understood with prospect theory’s value function. We examine how people feel about their progress on two goals (i.e., academics and fitness). What happens when they achieve one goal but fail to reach another? In four studies, we test the assumptions needed to explain hedonic reactions to progress along two goals. Loss aversion and diminishing sensitivity hold on each variable separately. However, we find violations of additivity in the integration of the emotions about outcomes. A success in one goal and a failure in another feel worse than the sum of the pleasure and pain associated with the gain and loss, respectively. The online appendix and data files are available at https://doi.org/10.1287/mnsc.2018.3097 . This paper was accepted by Elke Weber, judgment and decision making.
Research Interests:
Research Interests:
Research Interests: Psychology, Sentiment Analysis, Social Media, Medicine, Multidisciplinary, and 5 moreFootball, PLoS one, Bf, Hm, and League
Research Interests:
Research Interests:
Research Interests:
Research Interests:
... OUTSIDE VIEWS Kahneman and Tversky draw a sharp distinction between two modes offorecasting. ... at hand, and fo-cuses on classificatory variables with demonstrable predictive power. ... The decision-weighting function of prospect... more
... OUTSIDE VIEWS Kahneman and Tversky draw a sharp distinction between two modes offorecasting. ... at hand, and fo-cuses on classificatory variables with demonstrable predictive power. ... The decision-weighting function of prospect theory captures some of the strange ways ...
Research Interests:
Research Interests:
Research Interests:
When aggregating the probability estimates of many individuals to form a consensus probability estimate of an uncertain future event, it is common to combine them using a simple weighted average. Such aggregated probabilities correspond... more
When aggregating the probability estimates of many individuals to form a consensus probability estimate of an uncertain future event, it is common to combine them using a simple weighted average. Such aggregated probabilities correspond more closely to the real world if they are transformed by pushing them closer to 0 or 1. We explain the need for such transformations in terms of two distorting factors: The first factor is the compression of the probability scale at the two ends, so that random error tends to push the average probability toward 0.5. This effect does not occur for the median forecast, or, arguably, for the mean of the log odds of individual forecasts. The second factor—which affects mean, median, and mean of log odds—is the result of forecasters taking into account their individual ignorance of the total body of information available. Individual confidence in the direction of a probability judgment (high/low) thus fails to take into account the wisdom of crowds that ...
Research Interests:
Intentional harm is often a catalyst to action for victims and third party observers. Yet, harm may also be unintended. If there is a third party observer, and the victim does not know whether harm was intentional or accidental, would... more
Intentional harm is often a catalyst to action for victims and third party observers. Yet, harm may also be unintended. If there is a third party observer, and the victim does not know whether harm was intentional or accidental, would that observer reveal the offender’s intentions to the victim? In five studies, we investigated when and why third party observers reveal offenders’ intentions. We used an economic game in which participants observed a player being monetarily harmed either accidentally or intentionally. Third party observers were more likely to inform the victims when harm was accidental than intentional. We distinguish between two types of motives. First, emotional motives are desires to express one’s moral anger and empathy. Second, instrumental motives reflect desires to correct the victim’s impressions and ensure future fairness. Third party behavior is motivated by moral emotions of anger and empathy. Anger toward the offenders increases the likelihood that the observer will inform the victim when harm is intentional. Empathy for the victim increases the likelihood of informing the victim about intentional and accidental harm. Moreover, third party behavior is motivated by instrumental motives that reflect fairness concerns. By correcting the victim’s impression of the offender, the observer is ensuring fair future interactions between victim and offender. We discuss implications for forgiveness and management of conflict.