Development of a Crowd-Powered System Architecture for Knowledge Discovery in Scientific Domains
Authors: 
António Correia, 
Benjamim Fonseca, 
Hugo Paredes, Daniel Schneider, Shoaib JameelAuthors Info & Claims
António Correia, Benjamim Fonseca, Hugo Paredes, Daniel Schneider, Shoaib JameelAuthors Info & ClaimsPages 1372 - 1377
Abstract
A substantial amount of work is often overlooked due to the exponential rate of growth in global scientific output across all disciplines. Current approaches for addressing this issue are usually limited in scope and often restrict the possibility of obtaining multidisciplinary views in practice. To tackle this problem, researchers can now leverage an ecosystem of citizens, volunteers and crowd workers to perform complex tasks that are either difficult for humans and machines to solve alone. Motivated by the idea that human crowds and computer algorithms have complementary strengths, we present an approach where the machine will learn from crowd behavior in an iterative way. This approach is embodied in the architecture of SciCrowd, a crowd-powered human-machine hybrid system designed to improve the analysis and processing of large amounts of publication records. To validate the proposal’s feasibility, a prototype was developed and an initial evaluation was conducted to measure its robustness and reliability. We conclude this paper with a set of implications for design.
References
[1]
K. Li, J. Greenberg, and J. Dunic, “ Data objects and documenting scientific processes: An analysis of data events in biodiversity data papers,” J Assoc Inf Sci Technol , 2019 .
[2]
A. Basiri, M. Haklay, and Z. Gardner, The impact of biases in the crowdsourced trajectories on the output of data mining processes . in Proc. of the AGILE Conference on Geo-information science , 2018 .
[3]
D. Paine and C. P. Lee, “ Who has plots?: Contextualizing scientific software, practice, and visualizations,” in Proceedings of the ACM Conf. Human-Computer Interaction, CSCW , vol. 1, article 85, 2017 .
[4]
E. Law, K. Z. Gajos, A. Wiggins, M. L. Gray, and A. Williams, “ Crowdsourcing as a tool for research: Implications of uncertainty,” in Proceedings of the ACM Conference on Computer Supported Cooperative Work and Social Computing , 2017, pp. 1544 – 1561 .
[5]
M. Vukovic and C. Bartolini, “ Crowd-driven processes: state of the art and research challenges,” in Proceedings of the International Conference on Service-Oriented Computing , 2010, pp. 733 – 733 .
[6]
D. Paine, E. Sy, R. Piell, and C. P. Lee, “ Examining data processing work as part of the scientific data lifecycle: Comparing practices across four scientific research groups,” in Proc. iConference , 2015 .
[7]
M. Weiss, “ Crowdsourcing literature reviews in new domains,” Technology Innovation Management Review , 6, 2, 2016, pp. 5 – 14 .
[8]
M. Färber, A. Sampath, and A. Jatowt, “ PaperHunter: A system for exploring papers and citation contexts,” in Proceedings of the European Conference on Information Retrieval , 2019, pp. 246 – 250 .
[9]
A. Correia, D. Schneider, H. Paredes, and B. Fonseca, “ SciCrowd: Towards a hybrid, crowd-computing system for supporting research groups in academic settings,” in Proc. of the International Conference on Collaboration and Technology , 2018, pp. 34 – 41 .
[10]
H. Sauermann and C. Franzoni, “ Crowd science user contribution patterns and their implications,” National Academy of Sciences, vol. 112, no. 3, 2015, pp. 679 – 684 .
[11]
A. Correia, D. Schneider, B. Fonseca, and H. Paredes, “ Crowdsourcing and massively collaborative science: Systematic literature review and mapping study,” in Proc. of the International Conference on Collaboration and Technology , 2018, pp. 133 – 154 .
[12]
R. Tchoua, A. Ajith, Z. Hong, L. Ward, K. Chard, D. Audus, S. Patel, J. de Pablo, and I. Foster, “ Towards hybrid human-machine scientific information extraction,” New York Scientific Data Summit , 2018, 1 – 3 .
[13]
P. André, H. Zhang, J. Kim, L. Chilton, S. P. Dow, and R. C. Miller, “ Community clustering: Leveraging an academic crowd to form coherent conference sessions,” in Proceedings of the First AAAI Conference on Human Computation and Crowdsourcing , 2013 .
[14]
I. Peters, S. Haustein, and J. Terliesner, “ Crowdsourcing in article evaluation,” in Proceedings of the ACM International Conference on Web Science , 2011 .
[15]
K. M. Gregory, H. Cousijn, P. Groth, A. Scharnhorst, and S. Wyatt, “ Understanding data search as a socio-technical practice,” Journal of Information Science , 2019 .
[16]
R. Englund, S. Kottravel, and T. Ropinski, “ A crowdsourcing system for integrated and reproducible evaluation in scientific visualization,” in Proc. IEEE Pacific Visualization Symposium , 2016, pp. 40 – 47 .
[17]
A. T. Nguyen, A. Kharosekar, S. Krishnan, S. Krishnan, E. Tate, B. C. Wallace, and M. Lease, “ Believe it or not: Designing a human-AI partnership for mixed-initiative fact-checking,” in Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology , 2018, pp. 189 – 199 .
[18]
E. Horvitz, “ Principles of mixed-initiative user interfaces,” in Proceedings of the ACM CHI Conference on Human Factors in Computing Systems , 1999, pp. 159 – 166 .
[19]
Z. Dong, J. Lu, T. W. Ling, J. Fan, and Y. Chen, “ Using hybrid algorithmic-crowdsourcing methods for academic knowledge acquisition,” Cluster Computing , vol. 20, no. 4, 2017, pp. 3629 – 3641 .
[20]
M. Palacin-Silva and J. Porras, “ Shut up and take my environmental data! A study on ICT enabled citizen science practices, participation approaches and challenges,” EPiC Series in Computing , vol. 52, 2018, pp. 270 – 288 .
[21]
E. Krivosheev, F. Casati, and B. Benatallah, “ Crowd-based multi-predicate screening of papers in literature reviews,” arXiv:1803.09814 , 2018 .
[22]
V. Mohanty, D. Thames, S. Mehta, and K. Luther, “ Photo Sleuth: Combining human expertise and face recognition to identify historical portraits,” in Proceedings of the 24th International Conference on Intelligent User Interfaces , 2019, pp. 547 – 557 .
[23]
Y. Kim and A. Yoon, “ Scientists’ data reuse behaviors: A multilevel analysis,” Journal of the Association for Information Science and Technology vol. 68, no. 12, 2017, pp. 2709 – 2719 .
[24]
J. C. Chang, N. Hahn, A. Perer, and A. Kittur, “ SearchLens: Composing and capturing complex user interests for exploratory search,” in Proceedings of the 24th International Conference on Intelligent User Interfaces , 2019, pp. 498 – 509 .
[25]
D. Gurzick and W. G. Lutters, “ Towards a design theory for online communities,” in Proc. of the International Conf. on Design Science Research in Information Systems and Technology , 2009, p. 11 .
[26]
W. Willett, J. Heer, J. Hellerstein, and M. Agrawala, “ CommentSpace: Structured support for collaborative visual analysis,” in Proceedings of the ACM CHI Conference on Human Factors in Computing Systems , 2011, pp. 3131 – 3140 .
[27]
F. Daniel, P. Kucherbaev, C. Cappiello, B. Benatallah, and M. Allahbakhsh, “ Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions,” ACM Computing Surveys , vol. 51, no. 1, 2018 .
[28]
D. Ameller, M. Galster, P. Avgeriou, and X. Franch, “ A survey on quality attributes in service-based systems,” Software Quality Journal , vol. 24, no. 2, 2016, pp. 271 – 299 .
[29]
S. Greengard, “ Following the crowd,” Communications of the ACM , vol. 54, no. 2, 2011, pp. 20 – 22 .
[30]
S. Fu, A. Xu, X. Liu, H. Zhou, and R. Akkiraju, “ Challenge AI mind: A crowd system for proactive AI testing,” arXiv:1810.09030 , 2018 .
[31]
R. Sieber and V. Slonosky, “ Developing a flexible platform for crowdsourcing historical weather records,” Historical Methods: A Journal of Quantitative and Interdisciplinary History , 2019, pp. 1 – 14 .
[32]
H. Dickinson, L. Fortson, C. Scarlata, M. Beck, and M. Walmsley, “ Modeling with the crowd: Optimizing the human-machine partnership with Zooniverse,” arXiv:1903.07776 , 2019 .
[33]
I. Steinmacher, A. P. Chaves, and M. A. Gerosa, “ Awareness support in distributed software development: A systematic review and mapping of the literature,” Computer Supported Cooperative Work , vol. 22, nos. 2–3, 2013, pp. 113 – 158 .
[34]
J. Cheng and M. S. Bernstein, “ Flock: Hybrid crowd-machine learning classifiers,” in Proc. of the ACM Conference on Computer Supported Cooperative Work and Social Computing , 2015, pp. 600 – 611 .
[35]
J. Yang, T. Drake, A. Damianou, and Y. Maarek, “ Leveraging crowdsourcing data for deep active learning an application: Learning intents in Alexa,” in Proceedings of the International World Wide Web Conference , 2018, pp. 23 – 32 .
[36]
H. Xia, C. Østerlund, B. McKernan, J. Folkestad, P. Rossini, O. Boichak, J. Robinson, K. Kenski, R. Myers, B. A. Clegg, and J. Stromer-Galley, “ TRACE: A stigmergic crowdsourcing platform for intelligence analysis,” in Proceedings of the Hawaii International Conference on System Sciences , 2019, pp. 440 – 448 .
[37]
S. Amershi, D. Weld, M. Vorvoreanu, A. Fourney, B. Nushi, P. Collisson, J. Suh, S. Iqbal, P. N. Bennett, K. Inkpen, J. Teevan, R. Kikin-Gil, and E. Horvitz, “ Guidelines for human-AI Interaction,” in Proc. ACM Conf. on Human Factors in Computing Systems , 2019 .
[38]
T. B. Sheridan and W. L. Verplank, Human and computer control of undersea teleoperators . Massachusetts Institute of Technology Cambridge Man-Machine Systems Lab, 1978 .
Index Terms
- Development of a Crowd-Powered System Architecture for Knowledge Discovery in Scientific Domains
Index terms have been assigned to the content through auto-classification.
Recommendations
Technology footprints in scientific discovery: Citation contexts of paper‐to‐patent citations
AbstractCitations between papers and patents indicate knowledge transfer between science and technology. This study discusses the function of paper‐to‐patent citations through citation contexts. 7,223 articles having at least one paper‐to‐patent ...
Crowd Development: The Interplay between Crowd Evaluation and Collaborative Dynamics in Wikipedia
Collaborative crowdsourcing is an increasingly common way of accomplishing work in our economy. Yet, we know very little about how the behavior of these crowds changes over time and how these dynamics impact their performance. In this paper, we take a ...
Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments
The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that ...
Comments
Information & Contributors
Information
Published In
October 2019
4424 pages
Copyright © 2019.
Publisher
IEEE Press
Publication History
Published: 01 October 2019
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 12 Jan 2025