Unmasking Dunning-Kruger Effect in Visual Reasoning & Judgment
Authors: 
Mengyu Chen, Yijun Liu, Emily WallAuthors Info & Claims
Mengyu Chen, Yijun Liu, Emily WallAuthors Info & ClaimsPages 743 - 753
Abstract
The Dunning-Kruger Effect (DKE) is a metacognitive phenomenon where low-skilled individuals tend to overestimate their competence while high-skilled individuals tend to underestimate their competence. This effect has been observed in a number of domains including humor, grammar, and logic. In this paper, we explore if and how DKE manifests in visual reasoning and judgment tasks. Across two online user studies involving (1) a sliding puzzle game and (2) a scatterplot-based categorization task, we demonstrate that individuals are susceptible to DKE in visual reasoning and judgment tasks: those who performed best underestimated their performance, while bottom performers overestimated their performance. In addition, we contribute novel analyses that correlate susceptibility of DKE with personality traits and user interactions. Our findings pave the way for novel modes of bias detection via interaction patterns and establish promising directions towards interventions tailored to an individual's personality traits. All materials and analyses are in supplemental materials: https://github.com/CAV-Lab/DKE_supplemental.git.
References
[1]
L. Battle and J. Heer. “Characterizing exploratory visual analysis: A literature review and evaluation of analytic provenance in tableau”. In Computer graphics forum, vol. 38, pp. 145–159. Wiley Online Library, 2019.
[2]
E. Blakey and S. Spence. Developing metacognition. ERIC Clearinghouse on Information and Technology, 1990.
[3]
E. T. Brown, A. Ottley, H. Zhao, Q. Lin, R. Souvenir, A. Endert, and R. Chang. Finding waldo: Learning about users from their interactions. IEEE Transactions on visualization and computer graphics, 20(12): pp. 1663–1672, 2014.
[4]
A. Brüngger, A. Marzetta, K. Fukuda, and J. Nievergelt. The parallel search bench zram and its applications. Annals of Operations Research, 90(0): pp. 45–63, 1999.
[5]
K. A. Burson, R. P. Larrick, and J. Klayman. Skilled or unskilled, but still unaware of it: how perceptions of difficulty drive miscalibration in relative comparisons. Journal of personality and social psychology, 90(1): p. 60, 2006.
[6]
Y.-C. Chen, C.-Y. Chao, and H.-T. Hou. Learning pattern recognition skills from games: Design of an online pattern recognition educational mobile game integrating algebraic reasoning scaffolding. Journal of Educational Computing Research, 61(6): pp. 1232–1251, 2023.
[7]
I. Cho, R. Wesslen, A. Karduni, S. Santhanam, S. Shaikh, and W. Dou. The anchoring effect in decision-making with visual analytics. In 2017 IEEE Conference on Visual Analytics Science and Technology (VAST), pp. 116–126. IEEE, Phoenix, Arizona, USA.
[8]
L. H. Christoph et al. The role of metacognitive skills in learning to solve problems. SIKS, 2006.
[9]
P. Cowley, L. Nowell, and J. Scholtz. Glass box: An instrumented infrastructure for supporting human interaction with information. In Proceedings of the 38th Annual Hawaii International Conference on System Sciences, pp. 296c–296c. IEEE, Hawaii, USA, 2005.
[10]
M. Danesi and M. Danesi. Puzzles and spatial reasoning. Ahmes' Legacy: Puzzles and the Mathematical Mind, pp. 105–125, 2018.
[11]
D. A. Davis, P. E. Mazmanian, M. Fordis, R. Van Harrison, K. E. Thorpe, and L. Perrier. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. Jama, 296(9): pp. 1094–1102, 2006.
[12]
J. Diamond and W. Evans. The correction for guessing. Review of educational research, 43(2): pp. 181–191, 1973.
[13]
E. Dimara, G. Bailly, A. Bezerianos, and S. Franconeri. Mitigating the attraction effect with visualizations. IEEE transactions on visualization and computer graphics, 25(1): pp. 850–860, 2019.
[14]
E. Dimara, S. Franconeri, C. Plaisant, A. Bezerianos, and P. Dragicevic. A task-based taxonomy of cognitive biases for information visualization. IEEE transactions on visualization and computer graphics, 26(2): pp. 1413–1432, 2020.
[15]
M. B. Donnellan, F. L. Oswald, B. M. Baird, and R. E. Lucas. The mini-ipip scales: tiny-yet-effective measures of the big five factors of personality. Psychological assessment, 18(2): p. 192, 2006.
[16]
W. Dou, D. H. Jeong, F. Stukes, W. Ribarsky, H. R. Lipford, and R. Chang. Recovering reasoning processes from user interactions. IEEE Computer Graphics and Applications, 29(3): pp. 52–61, 2009.
[17]
O. J. Dunn. Multiple comparisons among means. Journal of the American statistical association, 56(293): pp. 52–64, 1961.
[18]
D. Dunning. “The dunning-kruger effect: On being ignorant of one's own ignorance”. In Advances in experimental social psychology, vol. 44, pp. 247–296. Elsevier, 2011.
[19]
D. Dunning, K. Johnson, J. Ehrlinger, and J. Kruger. Why people fail to recognize their own incompetence. Current directions in psychological science, 12(3): pp. 83–87, 2003.
[20]
J. Ehrlinger, K. Johnson, M. Banner, D. Dunning, and J. Kruger. Why the unskilled are unaware: Further explorations of (absent) self-insight among the incompetent. Organizational behavior and human decision processes, 105(1): pp. 98–121, 2008.
[21]
M. Feng, E. Peck, and L. Harrison. Patterns and pace: Quantifying diverse exploration behavior with visualizations on the web. IEEE transactions on visualization and computer graphics, 25(1): pp. 501–511, 2019.
[22]
J. H. Flavell. Metacognition and cognitive monitoring: A new area of cognitive-developmental inquiry. American psychologist, 34(10): p. 906, 1979.
[23]
R. B. Frary. Formula scoring of multiple-choice tests (correction for guessing). Educational measurement: Issues and practice, 7(2): pp. 33–38, 1988.
[24]
G. Gigerenzer and D. G. Goldstein. Reasoning the fast and frugal way: models of bounded rationality. Psychological review, 103(4): p. 650, 1996.
[25]
G. Gigerenzer and P. M. Todd. Simple heuristics that make us smart. USA: Oxford University Press, 1999.
[26]
A. Goguey, C. Gutwin, Z. Chen, P. Suwanaposee, and A. Cockburn. Interaction pace and user preferences. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–14. ACM, Virtual Conference.
[27]
L. R. Goldberg. A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. Personality psychology in Europe, 7(1): pp. 7–28, 1999.
[28]
S. Gomez and D. Laidlaw. Modeling task performance for a crowd of users from interaction histories. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2465–2468. ACM, Austin, Texas, USA, 2012.
[29]
D. Gotz, S. Sun, and N. Cao. Adaptive contextualization: Combating bias during high-dimensional visualization and data selection. In Proceedings of the 21st International Conference on Intelligent User Interfaces, pp. 85–95. ACM, Sonoma, California, USA, 2016.
[30]
T. M. Green and B. Fisher. Towards the Personal Equation of Interaction: The impact of personality factors on visual analytics interface interaction. In 2010 IEEE Symposium on Visual Analytics Science and Technology, pp. 203–210. IEEE, Salt Lake City, Utah, USA.
[31]
P. E. Hart, N. J. Nilsson, and B. Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics, 4(2): pp. 100–107, 1968.
[32]
A. Heydari, H. Soltani, and M. Mohammadi-Nezhad. The effect of psy-chological skills training (goal setting, positive selftalk and imagery) on self-confidence of adolescent volleyball players. Pedagogics, psychology, medical-biological problems of physical training and sports, 22(4): pp. 189–194, 2018.
[33]
S. Huang. When peers are not peers and don't know it: The dunning-kruger effect and self-fulfilling prophecy in peer-review. Bioessays, 35(5): pp. 414–416, 2013.
[34]
J. Hullman, X. Qiao, M. Correll, A. Kale, and M. Kay. In pursuit of error: A survey of uncertainty visualization evaluation. IEEE transactions on visualization and computer graphics, 25(1): pp. 903–913, 2019.
[35]
D. Kahneman. Thinking, fast and slow. Macmillan, 2011.
[36]
D. Kahneman and S. Frederick. A model of heuristic judgment., 2005.
[37]
D. Keim, G. Andrienko, J.-D. Fekete, C. Görg, J. Kohlhammer, and G. Melançon, Visual analytics: Definition, process, and challenges. Springer, 2008. 7.
[38]
H. Kim, J. Choo, H. Park, and A. Endert. Interaxis: Steering scatterplot axes via observation-level interaction. IEEE transactions on visualization and computer graphics, 22(1): pp. 131–140, 2016.
[39]
Y.-S. Kim, K. Reinecke, and J. Hullman. Data through others' eyes: The impact of visualizing others' expectations on visualization interpretation. IEEE transactions on visualization and computer graphics, 24(1): pp. 760–769, 2018.
[40]
Y.-S. Kim, L. A. Walls, P. Krafft, and J. Hullman. A bayesian cognition approach to improve data visualization. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–14. ACM Glasgow, Scotland, UK.
[41]
J. Krueger and R. A. Mueller. Unskilled, unaware, or both? the better-than-average heuristic and statistical regression predict errors in estimates of own performance. Journal of personality and social psychology, 82(2): p. 180, 2002.
[42]
J. Kruger and D. Dunning. Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of personality and social psychology, 77(6): p. 1121, 1999.
[43]
S. Lee, S.-H. Kim, and B. C. Kwon. Vlat: Development of a visualization literacy assessment test. IEEE transactions on visualization and computer graphics, 23(1): pp. 551–560, 2017.
[44]
H. B. Mann and D. R. Whitney. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics, pp. 50–60, 1947.
[45]
M. Michael and W.-C. Lin. Experimental study of information measure and inter-intra class distance ratios on feature selection and orderings. IEEE Transactions on Systems, Man, and Cybernetics, (2): pp. 172–181, 1973.
[46]
A. V. Moere, M. Tomitsch, C. Wimmer, B. Christoph, and T. Grechenig. Evaluating the effect of style in information visualization. IEEE Transactions on Visualization and Computer Graphics, 18(12): pp. 2739–2748, 2012.
[47]
D. A. Moore and P. J. Healy. The trouble with overconfidence. Psychological review, 115(2): pp. 502–517, 2008.
[48]
A. Narechania, A. Coscia, E. Wall, and A. Endert. Lumos: Increasing Awareness of Analytic Behavior during Visual Data Analysis. IEEE Transactions on Visualization and Computer Graphics, 28(1): pp. 1009–1018, 2022.
[49]
C. North, R. Chang, A. Endert, W. Dou, R. May, B. Pike, and G. Fink. Analytic provenance: process+interaction+insight. In CHI'11 Extended Abstracts on Human Factors in Computing Systems, pp. 33–36. ACM, Vancouver, BC, Canada, 2011.
[50]
S. R. Pavel, M. F. Robertson, and B. T. Harrison. The dunning-kruger effect and siuc university's aviation students. Journal of Aviation Technology and Engineering, 2(1): p. 6, 2012.
[51]
K. Pearson. Vii. note on regression and inheritance in the case of two parents. in proceedings of the royal society of London, 58(347–352): pp. 240–242, 1895.
[52]
V. Peña-Araya, A. Bezerianos, and E. Pietriga. A comparison of geo-graphical propagation visualizations. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–14. ACM, Honolulu, Hawaii, USA.
[53]
E. D. Ragan, A. Endert, J. Sanyal, and J. Chen. Characterizing provenance in visualization and data analysis: an organizational framework of provenance types and purposes. IEEE transactions on visualization and computer graphics, 22(1): pp. 31–40, 2016.
[54]
M. Rahmani. Medical trainees and the dunning–kruger effect: when they don't know what they don't know. Journal of Graduate Medical Education, 12(5): pp. 532–534, 2020. 1.
[55]
B. W. Roberts, C. Lejuez, R. F. Krueger, J. M. Richards, and P. L. Hill. What is conscientiousness and how can it be assessed? Developmental psychology, 50(5): p. 1315, 2014.
[56]
C. Sanchez and D. Dunning. Overconfidence among beginners: Is a little learning a dangerous thing? Journal of personality and Social Psychology, 114(1): p. 10, 2018.
[57]
D. J. Sanchez and A. Speed. The impact of individual traits on domain task performance: Exploring the dunning-kruger effect. Technical report, Sandia National Lab.(SNL-NM), Albuquerque, NM (United States), 2020.
[58]
S. S. Shapiro and M. B. Wilk. An analysis of variance test for normality (complete samples). Biometrika, 52(3/4): pp. 591–611, 1965.
[59]
The Decision Lab. Dunning–kruger effect. [Online]. Available: https://thedecisionlab.com/biases/dunning-kruger-effect, 2021. Retrieved March 7, 2024.
[60]
E. R. Thompson. Development and validation of an international english big-five mini-markers. Personality and individual differences, 45(6): pp. 542–548, 2008.
[61]
A. Tversky and D. Kahneman. Judgment under uncertainty: Heuristics and biases. science, 185(4157): pp. 1124–1131, 1974.
[62]
K. Umbleja, M. Ichino, and H. Yaguchi. Improving symbolic data visualization for pattern recognition and knowledge discovery. Visual Informatics, 4(1): pp. 23–31, 2020.
[63]
S. VanderPlas and H. Hofmann. Spatial reasoning and data displays. IEEE Transactions on Visualization and Computer Graphics, 22(1): pp. 459–468, 2016.
[64]
E. Wall, A. Arcalgud, K. Gupta, and A. Jo. A markov model of users' interactive behavior in scatterplots. In 2019 IEEE Visualization Conference (VIS), pp. 81–85. IEEE.
[65]
E. Wall, L. Blaha, C. Paul, and A. Endert. A formative study of interactive bias metrics in visual analytics using anchoring bias. In IFIP Conference on Human-Computer Interaction, pp. 555–575. Springer, Paphos, Cyprus, 2019.
[66]
E. Wall, L. M. Blaha, L. Franklin, and A. Endert. Warning, bias may occur: A proposed approach to detecting cognitive bias in interactive visual analytics. In 2017 ieee conference on visual analytics science and technology (vast), pp. 104–115. IEEE, Phoenix, Arizona, USA, 2017.
[67]
E. Wall, L. M. Blaha, C. L. Paul, K. Cook, and A. Endert. Four perspectives on human bias in visual analytics. Cognitive biases in visualizations, pp. 29–42, 2018.
[68]
E. Wall, A. Narechania, A. Coscia, J. Paden, and A. Endert. Left, right, and gender: Exploring interaction traces to mitigate human biases. IEEE transactions on visualization and computer graphics, 28(1): pp. 966–975, 2022.
[69]
E. Wall, J. Stasko, and A. Endert. Toward a design space for mitigating cognitive bias in vis. In 2019 IEEE Visualization Conference (VIS), pp. 111–115. IEEE, Vancouver, BC, Canada.
[70]
C. Ware. Information Visualization: Perception for Design. San Francisco: Morgan Kaufmann Publishers Inc., 2nd ed., 2004.
[71]
W. Willett, J. Heer, and M. Agrawala. Scented widgets: Improving navigation cues with embedded visualizations. IEEE transactions on visualization and computer graphics, 13(6): pp. 1129–1136, 2007.
[72]
C. Xiong, L. Van Weelden, and S. Franconeri. The curse of knowledge in visual data communication. IEEE transactions on visualization and computer graphics, 26(10): pp. 3051–3062, 2020.
[73]
F. Yang, M. Hedayati, and M. Kay. Subjective probability correction for uncertainty representations. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–17. Hamburg, Germany.
[74]
J. S. Yi, Y. ah Kang, J. Stasko, and J. A. Jacko. Toward a deeper understanding of the role of interaction in information visualization. IEEE transactions on visualization and computer graphics, 13(6): pp. 1224–1231, 2007.
[75]
Z. Zhou, X. Wen, Y. Wang, and D. Gotz. Modeling and leveraging analytic focus during exploratory visual analysis. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, article no., pp. 1–15. ACM, Yokohama, Japan.
[76]
C. Ziemkiewicz, A. Ottley, R. J. Crouser, A. R. Yauilla, S. L. Su, W. Ribarsky, and R. Chang. How visualization layout relates to locus of control and other personality factors. IEEE transactions on visualization and computer graphics, 19(7): pp. 1109–1121, 2013.
Index Terms
- Unmasking Dunning-Kruger Effect in Visual Reasoning & Judgment
Index terms have been assigned to the content through auto-classification.
Recommendations
Exploring the impact of emotion on visual judgement
VAST '12: Proceedings of the 2012 IEEE Conference on Visual Analytics Science and Technology (VAST)Existing research suggests that individual personality differences can influence performance with visualizations. In addition to stable traits such as locus of control, research in psychology has found that temporary changes in affect (emotion) can ...
An exploration of psychological factors on emoticon usage and implications for judgement accuracy
Given the increasing use of online platforms, the current research comprised two studies examining links between personality and emoticon use: Study 1 explored the psychological factors associated with emoticon usage on different online platforms (Nź=...
Influencing visual judgment through affective priming
CHI '13: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsRecent research suggests that individual personality differences can influence performance with visualizations. In addition to stable personality traits, research in psychology has found that temporary changes in affect (emotion) can also significantly ...
Comments
Information & Contributors
Information
Published In
1077-2626 © 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Educational Activities Department
United States
Publication History
Published: 17 September 2024
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 14 Jan 2025