Cited By
View all- Böttcher LKlingebiel R(2025)Organizational Selection of InnovationOrganization Science10.1287/orsc.2023.1735736:1(387-410)Online publication date: 1-Jan-2025
The Crowd Classification Problem: : Social Dynamics of Binary-Choice Accuracy
Abstract
Decades of research suggest that information exchange in groups and organizations can reliably improve judgment accuracy in tasks such as financial forecasting, market research, and medical decision making. However, we show that improving the accuracy of numeric estimates does not necessarily improve the accuracy of decisions. For binary-choice judgments, also known as classification tasks—for example, yes/no or build/buy decisions—social influence is most likely to grow the majority vote share, regardless of the accuracy of that opinion. As a result, initially, inaccurate groups become increasingly inaccurate after information exchange, even as they signal stronger support. We term this dynamic the “crowd classification problem.” Using both a novel data set and a reanalysis of three previous data sets, we study this process in two types of information exchange: (1) when people share votes only, and (2) when people form and exchange numeric estimates prior to voting. Surprisingly, when people exchange numeric estimates prior to voting, the binary-choice vote can become less accurate, even as the average numeric estimate becomes more accurate. Our findings recommend against voting as a form of decision making when groups are optimizing for accuracy. For those cases where voting is required, we discuss strategies for managing communication to avoid the crowd classification problem. We close with a discussion of how our results contribute to a broader contingency theory of collective intelligence.
This paper was accepted by Lamar Pierce, organizations.
References
[1]
Aggarwal R, Lucey BM (2007) Psychological barriers in gold prices? Rev. Financial Econom. 16(2):217–230.
[2]
Almaatouq A, Rahimian MA, Alhajri A (2020a) When social influence promotes the wisdom of crowds. Preprint, submitted June 22, https://arxiv.org/abs/2006.12471.
[3]
Almaatouq A, Yin M, Watts D (2020b) Collective problem-solving of groups across tasks of varying complexity. Preprint, submitted January 29, https://psyarxiv.com/ra9qy.
[4]
Almaatouq A, Becker J, Houghton JP, Paton N, Watts DJ, Whiting ME (2021) Empirica: A virtual lab for high-throughput macro-level experiments. Behav. Res. Methods, ePub ahead of print March 29, https://doi.org/10.3758/s13428-020-01535-9.
[5]
Almaatouq A, Noriega-Campero A, Alotaibi A, Krafft PM, Moussaid M, Pentland A (2020c) Adaptive social networks promote the wisdom of crowds. Proc. Natl. Acad. Sci. USA 117(21):11379–11386.
[6]
Amipour P, Gray SA, Jetter AJ, Introne JE, Singer A, Arlinghaus R (2020) Wisdom of stakeholder crowds in complex social–ecological systems. Nature Sustainability 3(3):191–199.
[7]
Arlotto A, Chick SE, Gans N (2014) Optimal hiring and retention policies for heterogeneous workers who learn. Management Sci. 60(1):110–129.
[8]
Asch SE (1951) Effects of group pressure upon the modification and distortion of judgments. Guetzkow H, ed. Groups, Leadership, and Men: Research in Human Relations (Carnegie Press, Pittsburgh), 177–190.
[9]
Ashton RH (1986) Combining the judgments of experts: How many and which ones? Organ. Behav. Human Decision Processes 38(3):405–414.
[10]
Atanasov P, Rescober P, Stone E, Swift SA, Servan-Schreiber E, Tetlock P, Ungar L, Mellers B (2017) Distilling the wisdom of crowds: Prediction markets vs. prediction polls. Management Sci. 63(3):691–706.
[11]
Bazerman MH, Moore DA (1994) Judgment in Managerial Decision Making (Wiley, New York).
[12]
Bazerman MH, Curhan JR, Moore DA, Valley KL (2000) Negotiation. Annual Rev. Psych. 51(1):279–314.
[13]
Becker J, Almaatouq A, Horvat A (2020) Network structures of collective intelligence: The contingent benefits of group discussion. Preprint, submitted September 15, https://arxiv.org/abs/2009.07202.
[14]
Becker J, Brackbill D, Centola D (2017) Network dynamics of social influence in the wisdom of crowds. Proc. Natl. Acad. Sci. USA 114(26):E5070–E5076.
[15]
Budescu DV, Chen E (2014) Identifying expertise to extract the wisdom of crowds. Management Sci. 61(2):267–280.
[16]
Burghardt K, Rand W, Girvan M (2019) Inferring models of opinion dynamics from aggregated jury data. PLoS One 14(7):e0218312.
[17]
Cohen MD, March JG, Olsen JP (1972) A garbage can model of organizational choice. Admin. Sci. Quart. 17(1):1–25.
[18]
Coleman JS (1966) Foundations for a theory of collective decisions. Amer. J. Sociol. 71(6):615–627.
[19]
Collins R (2014) Interaction ritual chains and collective effervescence. von Scheve C, Salmella M, eds. Collective Emotions, Series in Affective Science (Oxford University Press, Oxford, UK), 299–311.
[20]
Cowgill B, Zitzewitz E (2015) Corporate prediction markets: Evidence from google, ford, and firm x. Rev. Econom. Stud. 82(4):1309–1341.
[21]
Csaszar FA, Eggers JP (2013) Organizational decision making: An information aggregation view. Management Sci. 59(10):2257–2277.
[22]
Cyert RM, March JG (1963) A Behavioral Theory of the Firm (Prentice-Hall, Englewood Cliffs, NJ).
[23]
Da Z, Huang X (2020) Harnessing the wisdom of crowds. Management Sci. 66(5):1847–1867.
[24]
Dalkey N, Helmer O (1963) An experimental application of the delphi method to the use of experts. Management Sci. 9(3):458–467.
[25]
DeGroot MH (1974) Reaching a consensus. J. Amer. Statist. Assoc. 69(345):118–121.
[26]
Eichler HG, Kong SX, Gerth WC, Mavros P, Jönsson B (2004) Use of cost-effectiveness analysis in health-care resource allocation decision-making: How are cost-effectiveness thresholds expected to emerge? Value Health 7(5):518–528.
[27]
Farrell S (2011) Social influence benefits the wisdom of individuals in the crowd. Proc. Natl. Acad. Sci. USA 108(36):E625.
[28]
Fisher D, Charkoudian L (2008) Mediation Guide (Community Mediation Maryland).
[29]
Fisher R, Ury WL, Patton B (2011) Getting to Yes: Negotiating Agreement Without Giving In (Penguin).
[30]
Frey V, van de Rijt A (2020) Social influence undermines the wisdom of the crowd in sequential decision making. Management Sci., ePub ahead of print October 8, https://doi.org/10.1287/mnsc.2020.3713.
[31]
Gardikiotis A, Martin R, Hewstone M (2005) Group consensus in social influence: Type of consensus information as a moderator of majority and minority influence. Personality Soc. Psych. Bull. 31(9):1163–1174.
[32]
Golub B, Jackson MO (2010) Naive learning in social networks and the wisdom of crowds. Amer. Econom. J. Microeconom. 2(1):112–149.
[33]
Gürçay B, Mellers BA, Baron J (2015) The power of social influence on estimation accuracy. J. Behav. Decision Making 28(3):250–261.
[34]
Hartnett N, Kennedy R, Sharp B, Greenacre L (2016) Marketers’ intuitions about the sales effectiveness of advertisements. J. Marketing Behav. 2(2–3):177–194.
[35]
Hastie R (1986) Experimental evidence on group accuracy. Grofman B, Owen G, eds. Information Pooling and Group Decision Making, Decision Research, vol. 2 (Emerald, Bingley, UK), 129–157.
[36]
Hastie R, Kameda T (2005) The robust beauty of majority rules in group decisions. Psych. Rev. 112(2):494–508.
[37]
Hogarth RM (1978) A note on aggregating opinions. Organ. Behav. Human Performance 21(1):40–46.
[38]
Holman D, Chissick C, Totterdell P (2002) The effects of performance monitoring on emotional labor and well-being in call centers. Motivation Emotion 26(1):57–81.
[39]
Hong H, Du Q, Wang G, Fan W, Xu D (2016) Crowd wisdom: The impact of opinion diversity and participant independence on crowd performance. 22nd Amer. Conf. Inform. Systems AMCIS2016.
[40]
Jansen WJ, Jin X, de Winter JM (2016) Forecasting and nowcasting real GDP: Comparing statistical models and subjective forecasts. Internat. J. Forecasting 32(2):411–436.
[41]
Jayles B, Kim H-r, Escobedo R, Cezera S, Blanchet A, Kameda T, Sire C, Theraulaz G (2017) How social information can improve estimation accuracy in human groups. Proc. Natl. Acad. Sci. USA 114(47):12620–12625.
[42]
Jørgensen M (2004) A review of studies on expert estimation of software development effort. J. Systems Software 70(1–2):37–60.
[43]
Kao AB, Berdahl AM, Hartnett AT, Lutz MJ, Bak-Coleman JB, Ioannou CC, Giam X, Couzin ID (2018) Counteracting estimation bias and social influence to improve the wisdom of crowds. J. Roy. Soc. Interface 15(141):20180130.
[44]
Kinney W, Burgstahler D, Martin R (2002) Earnings surprise “materiality” as measured by stock returns. J. Accounting Res. 40(5):1297–1329.
[45]
Koriat A (2012) When are two heads better than one and why? Sci. 336(6079):360–362.
[46]
Krauss C (2008) Oil hits $100 a barrel for the first time. New York Times (January 2), https://www.nytimes.com/2008/01/02/business/02cnd-oil.html.
[47]
Kurvers RH, Herzog SM, Hertwig R, Krause J, Carney PA, Bogart A, Argenziano G, Zalaudek I, Wolf M (2016) Boosting medical diagnostics by pooling independent judgments. Proc. Natl. Acad. Sci. USA 113(31):8777–8782.
[48]
Lazer D, Friedman A (2007) The network structure of exploration and exploitation. Admin. Sci. Quart. 52(4):667–694.
[49]
Lorenz J, Rauhut H, Schweitzer F, Helbing D (2011) How social influence can undermine the wisdom of crowd effect. Proc. Natl. Acad. Sci. USA 108(22):9020–9025.
[50]
Madirolas G, de Polavieja GG (2015) Improving collective estimations using resistance to social influence. PLOS Comput. Biol. 11(11):e1004594.
[51]
Malone TW, Bernstein MS (2015) Handbook of Collective Intelligence (MIT Press, Cambridge, MA).
[52]
Mannes AE (2009) Are we wise about the wisdom of crowds? The use of group judgments in belief revision. Management Sci. 55(8):1267–1279.
[53]
Mannes AE, Soll JB, Larrick RP (2014) The wisdom of select crowds. J. Personality Soc. Psych. 107(2):276–299.
[54]
March JG (1991) How decisions happen in organizations. Human Comput. Interaction 6(2):95–117.
[55]
Medvec VH, Galinsky AD (2005) Putting more on the table: How making multiple offers can increase the final value of the deal. HBS Negotiation Newsletter 8:4–6.
[56]
Minson JA, Mueller JS, Larrick RP (2018) The contingent wisdom of dyads: When discussion enhances vs. undermines the accuracy of collaborative judgments. Management Sci. 64(9):4177–4192.
[57]
Mossel E, Tamuz O (2017) Opinion exchange dynamics. Probab. Surv. 14:155–204.
[58]
Moore P (2019) The quantified workplace. Medium (January 14) https://medium.com/field-guide-to-the-future-of-work/the-quantified-workplace-3874b53353d6.
[59]
Navajas J, Niella T, Garbulsky G, Bahrami B, Sigman M (2018) Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds. Nature Human Behav. 2:126–132.
[60]
Nofer M, Hinz O (2014) Are crowds on the internet wiser than experts? The case of a stock prediction community. J. Bus. Econom. 84(3):303–338.
[61]
Page SE (2007) The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies (Princeton University Press, Princeton, NJ).
[62]
Palley AB, Soll JB (2019) Extracting the wisdom of crowds when information is shared. Management Sci. 65(5):2291–2309.
[63]
Parayitam S, Dooley RS (2007) The relationship between conflict and decision outcomes: Moderating effects of cognitive- and affect-based trust in strategic decision-making teams. Internat. J. Conflict Management 18(1):42–73.
[64]
Pierce L, Snow DC, McAfee A (2015) Cleaning house: The impact of information technology monitoring on employee theft and productivity. Management Sci. 61(10):2299–2319.
[65]
Rivera LA (2012) Hiring as cultural matching: The case of elite professional service firms. Amer. Sociol. Rev. 77(6):999–1022.
[66]
Schnusenberg O (2006) The stock market behaviour prior and subsequent to new highs. Appl. Financial Econom. 16(6):429–438.
[67]
Sherif M (1935) A study of some social factors in perception. Arch. Psych. 187:60.
[68]
Shore J, Bernstein E, Lazer D (2015) Facts and figuring: An experimental investigation of network structure and performance in information and solution spaces. Organ. Sci. 26(5):1432–1446.
[69]
Straub VJ, Tsvetkova M, Yasseri T (2020) The cost of coordination can exceed the benefit of collaboration in performing complex tasks. Preprint, submitted September 23, https://arxiv.org/abs/2009.11038a.
[70]
Stroebe W, Nijstad BA, Rietzschel EF (2010) Chapter four—beyond productivity loss in brainstorming groups: The evolution of a question. Adv. Experiment. Soc. Psych. 43:157–203.
[71]
Surowiecki J (2004) The Wisdom of Crowds (Anchor Press, New York).
[72]
Ven AHVD, Delbecq AL (1974) The effectiveness of nominal, delphi, and interacting group decision making processes. Acad. Management J. 17(4):605–621.
[73]
Wolf M, Krause J, Carney PA, Bogart A, Kurvers RHJM (2015) Collective intelligence meets medical decision-making: The collective outperforms the best radiologist. PLoS One 10(8):e0134269.
Recommendations
Approximating the crowd
The problem of "approximating the crowd" is that of estimating the crowd's majority opinion by querying only a subset of it. Algorithms that approximate the crowd can intelligently stretch a limited budget for a crowdsourcing task. We present an ...
A Heuristic Model of Network-Based Group Decision Making for E-Services
ITNG '09: Proceedings of the 2009 Sixth International Conference on Information Technology: New GenerationsDecision making in groups can ease personal biases which enables the participants discuss, argue and coordinate the ideas of coalitions. The final outcome of a group decision making process reaches a consensus decision. However, using one particular ...
Boosting the Wisdom of Crowds Within a Single Judgment Problem: Weighted Averaging Based on Peer Predictions
A combination of point estimates from multiple judges often provides a more accurate aggregate estimate than a point estimate from a single judge, a phenomenon called “the wisdom of crowds.” However, if the judges use shared information when forming their ...
Comments
Information & Contributors
Information
Published In
Copyright © 2021, INFORMS.
Publisher
INFORMS
Linthicum, MD, United States
Publication History
Published: 01 May 2022
Accepted: 22 April 2021
Received: 19 March 2019
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 03 Feb 2025
Other Metrics
Citations
Cited By
View all- Böttcher LKlingebiel R(2025)Organizational Selection of InnovationOrganization Science10.1287/orsc.2023.1735736:1(387-410)Online publication date: 1-Jan-2025