research-article

Public Access

Comparing Different Sensemaking Approaches for Large-Scale Ideation

Authors:

Steven P. DowAuthors Info & Claims

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

Pages 2717 - 2728

https://doi.org/10.1145/2858036.2858178

Published: 07 May 2016 Publication History

Abstract

Large-scale idea generation platforms often expose ideators to previous ideas. However, research suggests people generate better ideas if they see abstracted solution paths (e.g., descriptions of solution approaches generated through human sensemaking) rather than being inundated with all prior ideas. Automated and semi-automated methods can also offer interpretations of earlier ideas. To benefit from sensemaking in practice with limited resources, ideation platform developers need to weigh the cost-quality tradeoffs of different methods for surfacing solution paths. To explore this, we conducted an online study where 245 participants generated ideas for two problems in one of five conditions: 1) no stimuli, 2) exposure to all prior ideas, or solution paths extracted from prior ideas using 3) a fully automated workflow, 4) a hybrid human-machine approach, and 5) a fully manual approach. Contrary to expectations, human-generated paths did not improve ideation (as meas-ured by fluency and breadth of ideation) over simply showing all ideas. Machine-generated paths sometimes significantly improved fluency and breadth of ideation over no ideas (although at some cost to idea quality). These findings suggest that automated sensemaking can improve idea generation, but we need more research to understand the value of human sensemaking for crowd ideation.

References

[1]

Teresa M. Amabile. 1983. The social psychology of creativity: A componential conceptualization. Journal of personality and social psychology 45, 2: 357.

[2]

Paul André, Aniket Kittur, and Steven P. Dow. 2014. Crowd Synthesis: Extracting Categories and Clusters from Complex Data. Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing, ACM, 989--998. http://doi.org/10.1145/2531602.2531653

Digital Library

[3]

Paul André, Haoqi Zhang, Juho Kim, Lydia Chilton, Steven P. Dow, and Robert C. Miller. 2013. Community clustering: Leveraging an academic crowd to form coherent conference sessions. First AAAI Conference on Human Computation and Crowdsourcing.

[4]

Brian P. Bailey and Eric Horvitz. 2010. What's Your Idea?: A Case Study of a Grassroots Innovation Pipeline Within a Large Software Company. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 2065--2074. http://doi.org/10.1145/1753326.1753641

Digital Library

[5]

Hugh Beyer and Karen Holtzblatt. 1997. Contextual design: defining customer-centered systems. Elsevier.

Digital Library

[6]

David M. Blei. 2012. Probabilistic topic models. Communications of the ACM 55, 4: 77--84. http://doi.org/10.1145/2133806.2133826

Digital Library

[7]

Kevin J. Boudreau and Karim R. Lakhani. 2013. Using the crowd as an innovation partner. Harvard Business Review 91, 4: 60--69.

[8]

Kevin J. Boudreau and Karim R. Lakhani. 2015. Open disclosure of innovations, incentives and follow-on reuse: Theory on processes of cumulative innovation and a field experiment in computational biology. Research Policy 44, 1: 4--19. http://doi.org/10.1016/j.respol.2014.08.001

[9]

Jonathan Chang, Sean Gerrish, Chong Wang, Jordan L. Boyd-graber, and David M. Blei. 2009. Reading tea leaves: How humans interpret topic models. Advances in neural information processing systems, 288--296.

Digital Library

[10]

Joel Chan, Steven C. Dang, and Steven P. Dow. 2016. Improving crowd innovation with expert facilitation. Proceedings of the ACM Conference on Computer Supported Cooperative Work & Social Computing. http://doi.org/10.1145/2818048.2820023

Digital Library

[11]

Joel Chan and Christian Schunn. 2015. The impact of analogies on creative concept generation: Lessons from an in vivo study in engineering design. Cognitive Science 39, 1: 126--155. http://doi.org/10.1111/cogs.12127

[12]

William G. Chase and Herbert A. Simon. 1973. The mind's eye in chess. In Visual Information Processing, William G. Chase (ed.). NY, NY, 215--281.

[13]

Duen Horng Chau, Aniket Kittur, Jason I. Hong, and Christos Faloutsos. 2011. Apolo: Making Sense of Large Network Data by Combining Rich User Interaction and Machine Learning. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 167--176. http://doi.org/10.1145/1978942.1978967

Digital Library

[14]

Lydia B. Chilton, Greg Little, Darren Edge, Daniel S. Weld, and James A. Landay. 2013. Cascade: Crowdsourcing taxonomy creation. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 1999--2008. http://doi.org/10.1145/2470654.2466265

Digital Library

[15]

Jacob Cohen. 1988. Statistical power analysis for the behavioral sciences. Psychology Press, Hillsdale, N.J.

[16]

Nelson Cowan. 2000. The magical number 4 in shortterm memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences 24: 87--185.

[17]

Darren W. Dahl and Page Moreau. 2002. The Influence and value of analogical thinking during new product ideation. Journal of Marketing Research 39, 1: 47--60.

[18]

Scott Deerwester, Susan T. Dumais, Geroge W. Furnas, and Thomas K. Landauer. 1990. Indexing by Latent Semantic Analysis. Journal of the American Society for Information Science 41, 6: 1990.

[19]

Darleen M. DeRosa, Carter L. Smith, and Donald A. Hantula. 2007. The medium matters: Mining the longpromised merit of group interaction in creative idea generation tasks in a meta-analysis of the electronic group brainstorming literature. Computers in Human Behavior 23, 3: 1549--1581. http://doi.org/10.1016/j.chb.2005.07.003

Digital Library

[20]

J. R. Duflou and P. Verhaegen. 2011. Systematic innovation through patent based product aspect analysis. CIRP Annals - Manufacturing Technology 60, 1: 203--206.

[21]

C. Eckert and M. Stacey. 1998. Fortune Favours Only the Prepared Mind: Why Sources of Inspiration are Essential for Continuing Creativity. Creativity and Innovation Management 7, 1: 1--12.

[22]

Antonio Ferreira, Valeria Herskovic, and Pedro Antunes. 2008. Attention-Based Management of Information Flows in Synchronous Electronic Brainstorming. In Groupware: Design, Implementation, and Use, Robert O. Briggs, Gert-Jan -. J. Vreede and Aaron S. Read (eds.). Springer-Verlag, Berlin, Heidelberg, 1--16. Retrieved from http://dx.doi.org/10.1007/978--3540--92831--7_1

Digital Library

[23]

Eve Forster and Kevin N. Dunbar. 2009. Creativity evaluation through Latent Semantic Analysis. Proceedings of the 31st Annual Conference of the Cognitive Science Society.

[24]

Katherine Fu, Joel Chan, Jonathan Cagan, Kenneth Kotovsky, Christian Schunn, and Kristin Wood. 2013. The Meaning of Near and Far: The Impact of Structuring Design Databases and the Effect of Distance of Analogy on Design Output. Journal of Mechanical Design 135, 2: 021007. http://doi.org/10.1115/1.4023158

[25]

M. L. Gick and K. J. Holyoak. 1983. Schema induction and analogical transfer. Cognitive Psychology 15, 1: 1--38.

[26]

Ryan G. Gomes, Peter Welinder, Andreas Krause, and Pietro Perona. 2011. Crowdclustering. Advances in Neural Information Processing Systems 24.

[27]

Joy P. Guilford. 1950. Creativity. American Psychologist 5: 444--454.

[28]

Joy P. Guilford. 1956. The structure of intellect. Psychological Bulletin 53, 4: 267--293. http://doi.org/10.1037/h0040755

[29]

Raja Gumienny, Steven Dow, Matthias Wenzel, Lutz Gericke, and Christoph Meinel. 2015. Tagging User Research Data: How to Support the Synthesis of Information in Design Teams. In Design Thinking Research, Hasso Plattner, Christoph Meinel and Larry Leifer (eds.). Springer International Publishing, 169--191.

[30]

J Isaiah Harbison and Henk Haarmann. 2014. Automated scoring of originality using semantic representations. Proceedings of the 36th Annual Conference of the Cognitive Science Society.

[31]

Beth A. Hennessey and Teresa M. Amabile. 2010. Creativity. Annual Review of Psychology 61: 569--98. http://doi.org/10.1146/annurev.psych.093008.100416

[32]

Scarlett R. Herring, Chia-Chen Chang, Jesse Krantzler, and Brian P. Bailey. 2009. Getting inspired!: understanding how and why examples are used in creative design practice. Proceedings of the 27th international conference on Human factors in computing systems, ACM, 87--96. http://doi.org/10.1145/1518701.1518717

Digital Library

[33]

Keith J. Holyoak and Paul Thagard. 1996. Mental leaps: Analogy in creative thought. Cambridge, MA.

Digital Library

[34]

David G. Jansson and Steven M. Smith. 1991. Design fixation. Design Studies 12, 1: 3--11.

[35]

Elahe Javadi and Wai-Tat -. T. Fu. 2011. Idea Visibility, Information Diversity, and Idea Integration in Electronic Brainstorming. Proceedings of the 6th International Conference on Foundations of Augmented Cognition: Directing the Future of Adaptive Systems, Springer-Verlag, 517--524.

Digital Library

[36]

Elahe Javadi, Joseph Mahoney, and Judith Gebauer. 2013. The impact of user interface design on idea integration in electronic brainstorming: an attention-based view. Journal of the Association for Information Systems 14, 1: 1--21.

[37]

Mark Klein and Gregorio Convertino. 2015. A Roadmap for Open Innovation Systems. Journal of Social Media for Organizations 2, 1: 1.

[38]

Jon Kolko. 2011. Exposing the magic of design: A practitioner's guide to the methods and theory of synthesis. Oxford University Press.

[39]

Adriana Kovashka and Kristen Grauman. 2014. Discovering Shades of Attribute Meaning with the Crowd. Third International Workshop on Parts and Attributes.

[40]

Filip Krynicki. 2014. Methods and models for quantitative analysis of crowd brainstorming. Master's thesis. University of Waterloo, Waterloo, Ontario, Canada.

[41]

Chinmay Kulkarni, Steven P. Dow, and Scott R. Klemmer. 2012. Early and repeated exposure to examples improves creative work. Proceedings of the 34th Annual Meeting of the Cognitive Science Society.

[42]

Thomas K. Landauer, Peter W. Foltz, and Darrell Laham. 1998. An introduction to latent semantic analysis. Discourse Processes 25, 2: 259--284.

[43]

Brian Lee, Savil Srivastava, Ranjitha Kumar, Ronen Brafman, and Scott R. Klemmer. 2010. Designing with Interactive Example Galleries. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 2257--2266. http://doi.org/10.1145/1753326.1753667

Digital Library

[44]

Julie S. Linsey, Ian Tseng, Katherine Fu, Jonathan Cagan, Kristin L. Wood, and Christian D. Schunn. 2010. A Study of Design Fixation, Its Mitigation and Perception in Engineering Design Faculty. Journal of Mechanical Design 132, 4: 041003.

[45]

Lan Luo and Olivier Toubia. 2015. Improving Online Idea Generation Platforms and Customizing the Task Structure Based on Consumers' Domain Specific Knowledge. Journal of Marketing. http://doi.org/10.1509/jm.13.0212

[46]

Douglas L. Medin. 1989. Concepts and conceptual structure. American Psychologist 44, 12: 1469--1481.

[47]

George A. Miller. 1956. The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychological Review 63, 2: 81--97. http://doi.org/10.1037/h0043158

[48]

Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014) 12: 1532--1543.

[49]

Bruce A. Reinig, Robert O. Briggs, and Jay F. Nunamaker. 2007. On the Measurement of Ideation Quality. Journal of Management Information Systems 23, 4: 143--161. http://doi.org/10.2753/MIS0742-

Digital Library

[50]

Mark A. Runco. 2004. Creativity. Annual Review of Psychology 55: 657--687.

[51]

R. Keith Sawyer. 2012. Explaining creativity: the science of human innovation. Oxford University Press, NY.

[52]

Jamie J. Shah, Noe Vargas-Hernandez, and Steven M. Smith. 2003. Metrics for measuring ideation effectiveness. Design Studies 24, 2: 111--134.

[53]

Pao Siangliulue, Kenneth C. Arnold, Krzysztof Z. Gajos, and Steven P. Dow. 2015. Toward Collaborative Ideation at Scale: Leveraging Ideas from Others to Generate More Creative and Diverse Ideas. Proceedings of the ACM Conference on Computer Supported Cooperative Work & Social Computing. http://doi.org/10.1145/2675133.2675239

Digital Library

[54]

Pao Siangliulue, Joel Chan, Kzryzstof Gajos, and Steven P. Dow. 2015. Providing timely examples improves the quantity and quality of generated ideas. Proceedings of the ACM Conference on Creativity and Cognition. http://doi.org/10.1145/2757226.2757230

Digital Library

[55]

Ut Na Sio, Kenneth Kotovsky, and Jonathan Cagan. 2015. Fixation or inspiration? A meta-analytic review of the role of examples on design processes. Design Studies 39: 70--99. http://doi.org/10.1016/j.destud.2015.04.004

[56]

Steven M. Smith, Thomas B. Ward, and Jay S. Schumacher. 1993. Constraining effects of examples in a creative generation task. Memory & Cognition 21, 6: 837--45.

[57]

John Stasko, Carsten Görg, and Zhicheng Liu. 2008. Jigsaw: Supporting Investigative Analysis through Interactive Visualization. Information Visualization 7, 2: 118--132. http://doi.org/10.1057/palgrave.ivs.9500180

[58]

Omer Tamuz, Ce Liu, Serge Belongie, Ohad Shamir, and Adam Tauman Kalai. 2011. Adaptively learning the crowd kernel. arXiv preprint arXiv:1105.1033.

[59]

Christian Terwiesch and Yi Xu. 2008. Innovation contests, open innovation, and multiagent problem solving. Management science 54, 9: 1529--1543.

Digital Library

[60]

Thomas B. Ward, Merryl J. Patterson, and Cynthia M. Sifonis. 2004. The Role of Specificity and Abstraction in Creative Idea Generation. Creativity Research Journal 16, 1: 1--9.

[61]

Thomas N. Wisdom and Robert L. Goldstone. 2011. Innovation, Imitation, and Problem Solving in a Networked Group. Nonlinear Dynamics-Psychology and Life Sciences 15, 2: 229.

[62]

Jinfeng Yi, Rong Jin, Shaili Jain, Tianbao Yang, and Anil K. Jain. 2012. Semi-crowdsourced clustering: Generalizing crowd labeling by robust distance metric learning. Advances in Neural Information Processing Systems, 1772--1780.

[63]

Yisong Yue, Chong Wang, Khalid El-Arini, and Carlos Guestrin. 2014. Personalized Collaborative Clustering. Proceedings of the 23rd International Conference on World Wide Web, ACM, 75--84. http://doi.org/10.1145/2566486.2567991

Digital Library

[64]

Lixiu Yu, Aniket Kittur, and Robert E. Kraut. 2014. Distributed Analogical Idea Generation: Inventing with Crowds. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 1245--1254. http://doi.org/10.1145/2556288.2557371

Digital Library

[65]

James Zou, Kamalika Chaudhuri, and Adam Kalai. 2015. Crowdsourcing Feature Discovery via Adaptively Chosen Comparisons. Third AAAI Conference on Human Computation and Crowdsourcing.

Cited By

Choi DHong SPark JChung JKim J(2024)CreativeConnect: Supporting Reference Recombination for Graphic Design Ideation with Generative AIProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642794(1-25)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642794
Lee SLaw MHoffman G(2024)When and How to Use AI in the Design Process? Implications for Human-AI Design CollaborationInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2353451(1-16)Online publication date: 22-May-2024
https://doi.org/10.1080/10447318.2024.2353451
Lei KYu MLewellen MKu VLee D(2023)Compass: Supporting Large Group Mentorship in a Chat-Based UIProceedings of the ACM on Human-Computer Interaction10.1145/35794707:CSCW1(1-25)Online publication date: 16-Apr-2023
https://dl.acm.org/doi/10.1145/3579470
Show More Cited By

Index Terms

Comparing Different Sensemaking Approaches for Large-Scale Ideation
1. Human-centered computing
  1. Collaborative and social computing

Recommendations

Toward Collaborative Ideation at Scale: Leveraging Ideas from Others to Generate More Creative and Diverse Ideas
CSCW '15: Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing

A growing number of large collaborative idea generation platforms promise that by generating ideas together, people can create better ideas than any would have alone. But how might these platforms best leverage the number and diversity of contributors ...
Semantically Far Inspirations Considered Harmful?: Accounting for Cognitive States in Collaborative Ideation
C&C '17: Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition

Collaborative ideation systems can help people generate more creative ideas by exposing them to ideas different from their own. However, there are competing theoretical views on whether and when such exposure is helpful. Associationist theory suggests ...
CrowdMuse: An Adaptive Crowd Brainstorming System
UIST '18 Adjunct: Adjunct Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology

Online crowds, with their large numbers and diversity, show great potential for creativity, particularly during large-scale brainstorming sessions. Research has explored different ways of augmenting this creativity, such as showing ideators some form of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

May 2016

6108 pages

ISBN:9781450333627

DOI:10.1145/2858036

General Chairs:
Jofish Kaye
Yahoo
,
Allison Druin
University of Maryland / National Park Service
,
Program Chairs:
Cliff Lampe
University of Michigan
,
Dan Morris
Microsoft
,
Juan Pablo Hourcade
University of Iowa

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

CHI'16

Sponsor:

SIGCHI

CHI'16: CHI Conference on Human Factors in Computing Systems

May 7 - 12, 2016

California, San Jose, USA

Acceptance Rates

CHI '16 Paper Acceptance Rate 565 of 2,435 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
1,458
Total Downloads

Downloads (Last 12 months)216
Downloads (Last 6 weeks)37

Reflects downloads up to 06 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Choi DHong SPark JChung JKim J(2024)CreativeConnect: Supporting Reference Recombination for Graphic Design Ideation with Generative AIProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642794(1-25)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642794
Lee SLaw MHoffman G(2024)When and How to Use AI in the Design Process? Implications for Human-AI Design CollaborationInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2353451(1-16)Online publication date: 22-May-2024
https://doi.org/10.1080/10447318.2024.2353451
Lei KYu MLewellen MKu VLee D(2023)Compass: Supporting Large Group Mentorship in a Chat-Based UIProceedings of the ACM on Human-Computer Interaction10.1145/35794707:CSCW1(1-25)Online publication date: 16-Apr-2023
https://dl.acm.org/doi/10.1145/3579470
Palani SZhou YZhu SDow S(2022)InterWeave: Presenting Search Suggestions in Context Scaffolds Information Search and SynthesisProceedings of the 35th Annual ACM Symposium on User Interface Software and Technology10.1145/3526113.3545696(1-16)Online publication date: 29-Oct-2022
https://dl.acm.org/doi/10.1145/3526113.3545696
Hope TTamari RHershcovich DKang HChan JKittur AShahaf D(2022)Scaling Creative Inspiration with Fine-Grained Functional Aspects of IdeasProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517434(1-15)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517434
Cheng RDasgupta SHill B(2022)How Interest-Driven Content Creation Shapes Opportunities for Informal Learning in Scratch: A Case Study on Novices’ Use of Data StructuresProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502124(1-16)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3502124
Kim KDavis RCoppi ACattaneo ADillenbourg P(2022)Mixplorer: Scaffolding Design Space Exploration through Genetic Recombination of Multiple Peoples’ Designs to Support Novices’ CreativityProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501854(1-13)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3501854
Venkatagiri SGautam ALuther K(2021)CrowdSolveProceedings of the ACM on Human-Computer Interaction10.1145/34491925:CSCW1(1-30)Online publication date: 22-Apr-2021
https://dl.acm.org/doi/10.1145/3449192
Cox SWang YAbdul Avon der Weth CY. Lim BKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)Directed Diversity: Leveraging Language Embedding Distances for Collective Creativity in Crowd IdeationProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445782(1-35)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445782
Koh YHan SPark J(2021)A systematic process for generating new blockchain-service business model ideasService Business10.1007/s11628-021-00465-616:1(187-209)Online publication date: 2-Nov-2021
https://doi.org/10.1007/s11628-021-00465-6
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents