Perspectives on psychological science : a journal of the Association for Psychological Science, 2015
Across a wide range of tasks, research has shown that people make poor probabilistic predictions ... more Across a wide range of tasks, research has shown that people make poor probabilistic predictions of future events. Recently, the U.S. Intelligence Community sponsored a series of forecasting tournaments designed to explore the best strategies for generating accurate subjective probability estimates of geopolitical events. In this article, we describe the winning strategy: culling off top performers each year and assigning them into elite teams of superforecasters. Defying expectations of regression toward the mean 2 years in a row, superforecasters maintained high accuracy across hundreds of questions and a wide array of topics. We find support for four mutually reinforcing explanations of superforecaster performance: (a) cognitive abilities and styles, (b) task-specific skills, (c) motivation and commitment, and (d) enriched environments. These findings suggest that superforecasters are partly discovered and partly created-and that the high-performance incentives of tournaments hig...
Journal of experimental psychology. Applied, Jan 12, 2015
This article extends psychological methods and concepts into a domain that is as profoundly conse... more This article extends psychological methods and concepts into a domain that is as profoundly consequential as it is poorly understood: intelligence analysis. We report findings from a geopolitical forecasting tournament that assessed the accuracy of more than 150,000 forecasts of 743 participants on 199 events occurring over 2 years. Participants were above average in intelligence and political knowledge relative to the general population. Individual differences in performance emerged, and forecasting skills were surprisingly consistent over time. Key predictors were (a) dispositional variables of cognitive ability, political knowledge, and open-mindedness; (b) situational variables of training in probabilistic reasoning and participation in collaborative teams that shared information and discussed rationales (Mellers, Ungar, et al., 2014); and (c) behavioral variables of deliberation time and frequency of belief updating. We developed a profile of the best forecasters; they were bet...
This article extends psychological methods and concepts into a domain that is as profoundly conse... more This article extends psychological methods and concepts into a domain that is as profoundly consequential as it is poorly understood: intelligence analysis. We report findings from a geopolitical forecasting tournament that assessed the accuracy of more than 150,000 forecasts of 743 participants on 199 events occurring over 2 years. Participants were above average in intelligence and political knowledge relative to the general population. Individual differences in performance emerged, and forecasting skills were surprisingly consistent over time. Key predictors were (a) dispositional variables of cognitive ability, political knowledge, and open-mindedness; (b) situational variables of training in probabilistic reasoning and participation in collaborative teams that shared information and discussed rationales (Mellers, Ungar, et al., 2014); and (c) behavioral variables of deliberation time and frequency of belief updating. We developed a profile of the best forecasters; they were better at inductive reasoning, pattern detection, cognitive flexibility, and open-mindedness. They had greater understanding of geopolitics, training in probabilistic reasoning, and opportunities to succeed in cognitively enriched team environments. Last but not least, they viewed forecasting as a skill that required deliberate practice, sustained effort, and constant monitoring of current affairs.
Across a wide range of tasks, research has shown that people make poor probabilistic predictions ... more Across a wide range of tasks, research has shown that people make poor probabilistic predictions of future events. Recently, the U.S. Intelligence Community sponsored a series of forecasting tournaments designed to explore the best strategies for generating accurate subjective probability estimates of geopolitical events. In this article, we describe the winning strategy: culling off top performers each year and assigning them into elite teams of superforecasters. Defying expectations of regression toward the mean 2 years in a row, superforecasters maintained high accuracy across hundreds of questions and a wide array of topics. We find support for four mutually reinforcing explanations of superforecaster performance: (a) cognitive abilities and styles, (b) task-specific skills, (c) motivation and commitment, and (d) enriched environments. These findings suggest that superforecasters are partly discovered and partly created— and that the high-performance incentives of tournaments highlight aspects of human judgment that would not come to light in laboratory paradigms focused on typical performance.
This article extends psychological methods and concepts into a domain that is as profoundly conse... more This article extends psychological methods and concepts into a domain that is as profoundly consequen- tial as it is poorly understood: intelligence analysis. We report findings from a geopolitical forecasting tournament that assessed the accuracy of more than 150,000 forecasts of 743 participants on 199 events occurring over 2 years. Participants were above average in intelligence and political knowledge relative to the general population. Individual differences in performance emerged, and forecasting skills were surprisingly consistent over time. Key predictors were (a) dispositional variables of cognitive ability, political knowledge, and open-mindedness; (b) situational variables of training in probabilistic reasoning and participation in collaborative teams that shared information and discussed rationales (Mellers, Ungar, et al., 2014); and (c) behavioral variables of deliberation time and frequency of belief updating. We developed a profile of the best forecasters; they were better at inductive reasoning, pattern detection, cognitive flexibility, and open-mindedness. They had greater understanding of geopolitics, training in probabilistic reasoning, and opportunities to succeed in cognitively enriched team environments. Last but not least, they viewed forecasting as a skill that required deliberate practice, sustained effort, and constant monitoring of current affairs.
Abstract: Compared with minimum competency tests, curriculum-based external exit exams provide be... more Abstract: Compared with minimum competency tests, curriculum-based external exit exams provide better measures of students' achievement levels. Analysis of Third Mathematics and Science Study data shows that 13 year-olds from exit-exam countries are ahead of ...
Perspectives on psychological science : a journal of the Association for Psychological Science, 2015
Across a wide range of tasks, research has shown that people make poor probabilistic predictions ... more Across a wide range of tasks, research has shown that people make poor probabilistic predictions of future events. Recently, the U.S. Intelligence Community sponsored a series of forecasting tournaments designed to explore the best strategies for generating accurate subjective probability estimates of geopolitical events. In this article, we describe the winning strategy: culling off top performers each year and assigning them into elite teams of superforecasters. Defying expectations of regression toward the mean 2 years in a row, superforecasters maintained high accuracy across hundreds of questions and a wide array of topics. We find support for four mutually reinforcing explanations of superforecaster performance: (a) cognitive abilities and styles, (b) task-specific skills, (c) motivation and commitment, and (d) enriched environments. These findings suggest that superforecasters are partly discovered and partly created-and that the high-performance incentives of tournaments hig...
Journal of experimental psychology. Applied, Jan 12, 2015
This article extends psychological methods and concepts into a domain that is as profoundly conse... more This article extends psychological methods and concepts into a domain that is as profoundly consequential as it is poorly understood: intelligence analysis. We report findings from a geopolitical forecasting tournament that assessed the accuracy of more than 150,000 forecasts of 743 participants on 199 events occurring over 2 years. Participants were above average in intelligence and political knowledge relative to the general population. Individual differences in performance emerged, and forecasting skills were surprisingly consistent over time. Key predictors were (a) dispositional variables of cognitive ability, political knowledge, and open-mindedness; (b) situational variables of training in probabilistic reasoning and participation in collaborative teams that shared information and discussed rationales (Mellers, Ungar, et al., 2014); and (c) behavioral variables of deliberation time and frequency of belief updating. We developed a profile of the best forecasters; they were bet...
This article extends psychological methods and concepts into a domain that is as profoundly conse... more This article extends psychological methods and concepts into a domain that is as profoundly consequential as it is poorly understood: intelligence analysis. We report findings from a geopolitical forecasting tournament that assessed the accuracy of more than 150,000 forecasts of 743 participants on 199 events occurring over 2 years. Participants were above average in intelligence and political knowledge relative to the general population. Individual differences in performance emerged, and forecasting skills were surprisingly consistent over time. Key predictors were (a) dispositional variables of cognitive ability, political knowledge, and open-mindedness; (b) situational variables of training in probabilistic reasoning and participation in collaborative teams that shared information and discussed rationales (Mellers, Ungar, et al., 2014); and (c) behavioral variables of deliberation time and frequency of belief updating. We developed a profile of the best forecasters; they were better at inductive reasoning, pattern detection, cognitive flexibility, and open-mindedness. They had greater understanding of geopolitics, training in probabilistic reasoning, and opportunities to succeed in cognitively enriched team environments. Last but not least, they viewed forecasting as a skill that required deliberate practice, sustained effort, and constant monitoring of current affairs.
Across a wide range of tasks, research has shown that people make poor probabilistic predictions ... more Across a wide range of tasks, research has shown that people make poor probabilistic predictions of future events. Recently, the U.S. Intelligence Community sponsored a series of forecasting tournaments designed to explore the best strategies for generating accurate subjective probability estimates of geopolitical events. In this article, we describe the winning strategy: culling off top performers each year and assigning them into elite teams of superforecasters. Defying expectations of regression toward the mean 2 years in a row, superforecasters maintained high accuracy across hundreds of questions and a wide array of topics. We find support for four mutually reinforcing explanations of superforecaster performance: (a) cognitive abilities and styles, (b) task-specific skills, (c) motivation and commitment, and (d) enriched environments. These findings suggest that superforecasters are partly discovered and partly created— and that the high-performance incentives of tournaments highlight aspects of human judgment that would not come to light in laboratory paradigms focused on typical performance.
This article extends psychological methods and concepts into a domain that is as profoundly conse... more This article extends psychological methods and concepts into a domain that is as profoundly consequen- tial as it is poorly understood: intelligence analysis. We report findings from a geopolitical forecasting tournament that assessed the accuracy of more than 150,000 forecasts of 743 participants on 199 events occurring over 2 years. Participants were above average in intelligence and political knowledge relative to the general population. Individual differences in performance emerged, and forecasting skills were surprisingly consistent over time. Key predictors were (a) dispositional variables of cognitive ability, political knowledge, and open-mindedness; (b) situational variables of training in probabilistic reasoning and participation in collaborative teams that shared information and discussed rationales (Mellers, Ungar, et al., 2014); and (c) behavioral variables of deliberation time and frequency of belief updating. We developed a profile of the best forecasters; they were better at inductive reasoning, pattern detection, cognitive flexibility, and open-mindedness. They had greater understanding of geopolitics, training in probabilistic reasoning, and opportunities to succeed in cognitively enriched team environments. Last but not least, they viewed forecasting as a skill that required deliberate practice, sustained effort, and constant monitoring of current affairs.
Abstract: Compared with minimum competency tests, curriculum-based external exit exams provide be... more Abstract: Compared with minimum competency tests, curriculum-based external exit exams provide better measures of students' achievement levels. Analysis of Third Mathematics and Science Study data shows that 13 year-olds from exit-exam countries are ahead of ...
Uploads
Papers by Michael M Bishop
as it is poorly understood: intelligence analysis. We report findings from a geopolitical forecasting
tournament that assessed the accuracy of more than 150,000 forecasts of 743 participants on 199 events
occurring over 2 years. Participants were above average in intelligence and political knowledge relative
to the general population. Individual differences in performance emerged, and forecasting skills were
surprisingly consistent over time. Key predictors were (a) dispositional variables of cognitive ability,
political knowledge, and open-mindedness; (b) situational variables of training in probabilistic reasoning
and participation in collaborative teams that shared information and discussed rationales (Mellers, Ungar,
et al., 2014); and (c) behavioral variables of deliberation time and frequency of belief updating. We
developed a profile of the best forecasters; they were better at inductive reasoning, pattern detection,
cognitive flexibility, and open-mindedness. They had greater understanding of geopolitics, training in
probabilistic reasoning, and opportunities to succeed in cognitively enriched team environments. Last but
not least, they viewed forecasting as a skill that required deliberate practice, sustained effort, and constant
monitoring of current affairs.
as it is poorly understood: intelligence analysis. We report findings from a geopolitical forecasting
tournament that assessed the accuracy of more than 150,000 forecasts of 743 participants on 199 events
occurring over 2 years. Participants were above average in intelligence and political knowledge relative
to the general population. Individual differences in performance emerged, and forecasting skills were
surprisingly consistent over time. Key predictors were (a) dispositional variables of cognitive ability,
political knowledge, and open-mindedness; (b) situational variables of training in probabilistic reasoning
and participation in collaborative teams that shared information and discussed rationales (Mellers, Ungar,
et al., 2014); and (c) behavioral variables of deliberation time and frequency of belief updating. We
developed a profile of the best forecasters; they were better at inductive reasoning, pattern detection,
cognitive flexibility, and open-mindedness. They had greater understanding of geopolitics, training in
probabilistic reasoning, and opportunities to succeed in cognitively enriched team environments. Last but
not least, they viewed forecasting as a skill that required deliberate practice, sustained effort, and constant
monitoring of current affairs.