Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
SpringerLink
Log in

Hint: harnessing the wisdom of crowds for handling multi-phase tasks

  • S.I. : Neural Computing for IOT based Intelligent Healthcare Systems
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

The resourcefulness of crowdsourcing can be used to handle a wide range of complex macro-tasks, such as travel planning, translation, and software development. Multi-phase tasks are a type of macro-task that consists of several subtasks distributed across multiple sequential phases. Due to the recent work’s disregard for the task’s sequential correlation, it is difficult for them to handle multi-stage tasks effectively. This work bridges this gap. We call this novel approach Hint, which incorporates task design, pre hoc worker coordination, and post hoc crowd work coordination. Starting with the task interface design, Hint makes workers aware of the relationship between phases in order to improve their processing abilities. Second, pre hoc coordination of workers is to organize the workers to do the tasks to lower the monetary costs required to meet a specific quality standard. Third, post hoc coordination of crowd work is through a decision tree-based coordination strategy. Extensive tests are carried out on real-world datasets to validate the desirable qualities of the suggested mechanism.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Howe J (2006) The rise of crowdsourcing. Wired Mag 14(6):1–4

    Google Scholar 

  2. Kamar E, Hacker S, Horvitz E (2012) Combining human and machine intelligence in large-scale crowdsourcing. In: Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems, AAMAS’2012. International Foundation for Autonomous Agents and Multiagent Systems, pp. 467–474

  3. Lampinen A, Lutz C, Newlands G, Light A, Immorlica N (2018) Power struggles in the digital economy: platforms, workers, and markets. In: Companion of the 2018 ACM Conference on Computer Supported Cooperative Work and Social Computing, CSCW’18. Association for Computing Machinery, New York, NY, USA, pp. 417–423 https://doi.org/10.1145/3272973.3273004

  4. Quinn AJ, Bederson BB (2011) Human computation: a survey and taxonomy of a growing field. In: Proceedings of the International Conference on Human Factors in Computing Systems, CHI’2011, pp. 1403–1412

  5. Han T, Sun H, Song Y, Fang Y, Liu X (2016) Incorporating external knowledge into crowd intelligence for more specific knowledge acquisition. In: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI’2016, AAAI Press, pp. 1541–1547

  6. Von Ahn L, Maurer B, McMillen C, Abraham D, Blum M (2008) reCAPTCHA: human-based character recognition via web security measures. Science 321(5895):1465–1468

    Article  MathSciNet  MATH  Google Scholar 

  7. Teevan J, Iqbal ST, von Veh C (2016) Supporting collaborative writing with microtasks. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI’2016, ACM, pp. 2657–2668

  8. Stol KJ, Fitzgerald B (2014) Two’s company, three’s a crowd: a case study of crowdsourcing software development. In: Proceedings of the 36th International Conference on Software Engineering, ICSE’2012, ACM, pp. 187–198

  9. Anderson TJ, Soden R, Keegan B, Palen L, Anderson KM (2018) The crowd is the territory: assessing quality in peer-produced spatial data during disasters. Int J Hum Comput Interact. 34(4):295–310

    Article  Google Scholar 

  10. Gao J, Li Q, Zhao B, Fan W, Han J (2015) Truth discovery and crowdsourcing aggregation: a unified perspective. Proc VLDB Endow 8(12):2048–2049

    Article  Google Scholar 

  11. Li G, Wang J, Zheng Y, Franklin MJ (2016) Crowdsourced data management: a survey. IEEE Trans Knowl Data Eng 28(9):2296–2319

    Article  Google Scholar 

  12. Zhou D, Liu Q, Platt JC, Meek C (2014) Aggregating ordinal labels from crowds by minimax conditional entropy. In: Proceedings of the 31th International Conference on Machine Learning, ICML’2014, pp. 262–270

  13. Sheng VS, Provost F, Ipeirotis PG (2008) Get another label? improving data quality and data mining using multiple, noisy labelers. In: KDD2008, KDD’2008, ACM, pp. 614–622

  14. Li H (2015) Theoretical analysis and efficient algorithms for crowdsourcing. Ph.D. thesis, University of California, Berkeley, USA. http://www.escholarship.org/uc/item/9rd5255z

  15. Li H, Liu Q (2015) Cheaper and better: selecting good workers for crowdsourcing. In: Proceedings of the Third AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2015, November 8–11, 2015, San Diego, California, USA, pp. 20–21

  16. Sheshadri A, Lease M (2013) Square: a benchmark for research on computing crowd consensus. In: Hartman B, Horvitz E (eds.) HCOMP. AAAI

  17. Gaunt A, Borsa D, Bachrach Y (2016) Training neural nets to aggregate crowdsourced responses. In: Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence, UAI 2016, June 25–29, 2016, New York City, NY, USA. AUAI Press

  18. Lykourentzou I, Khan V, Papangelis K, Markopoulos P (2019) Macrotask crowdsourcing: an integrated definition. In: Khan V, Papangelis K, Lykourentzou I, Markopoulos P (eds) Macrotask Crowdsourcing - engaging the crowds to address complex problems, human-computer interaction series. Springer, New York, pp 1–13

    Google Scholar 

  19. Roper T, Dutka LM, Cobb S, Patel H (2019) Collaborative virtual environment to facilitate game design evaluation with children with ASC. Int J Hum Comput Interact 35(8):692–705

    Article  Google Scholar 

  20. Chen L, Shahabi C (2016) Spatial crowdsourcing: challenges and opportunities. IEEE Data Eng Bull 39(4):14–25

    Google Scholar 

  21. Dow S, Kulkarni A, Bunge B, Nguyen T, Klemmer S, Hartmann B (2011) Shepherding the crowd: managing and providing feedback to crowd workers. In: CHI’11 Extended Abstracts on Human Factors in Computing Systems, CHI’2011, ACM, pp. 1669–1674

  22. Fang Y, Sun H, Li G, Zhang R, Huai J (2016) Effective result inference for context-sensitive tasks in crowdsourcing. In: DASFAA2016, DASFAA’2016, pp. 33–48

  23. Ryskeldiev B, Igarashi T, Zhang J, Ochiai Y, Cohen M, Herder J (2018) Spotility: crowdsourced telepresence for social and collaborative experiences in mobile mixed reality. In: Companion of the 2018 ACM Conference on Computer Supported Cooperative Work and Social Computing, CSCW 2018, Jersey City, NJ, USA, November 03–07, 2018, pp. 373–376

  24. Sprinks J, Wardlaw J, Houghton R, Bamford S, Morley J (2017) Task workflow design and its impact on performance and volunteers’ subjective preference in virtual citizen science. Int J Hum Comput Stud 104:50–63

    Article  Google Scholar 

  25. Zhu Y, Yue S, Yu C, Shi Y (2017) Cept: collaborative editing tool for non-native authors. In: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, CSCW’2017, ACM, pp. 273–285

  26. LaToza TD, Lecce AD, Ricci F, Towne WB, van der Hoek A (2019) Microtask programming. IEEE Trans Softw Eng 45(11):1106–1124

    Article  Google Scholar 

  27. Chilton LB, Little G, Edge D, Weld DS, Landay JA (2013) Cascade: crowdsourcing taxonomy creation. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’2013, ACM, pp. 1999–2008

  28. Lotosh I, Milo T, Novgorodov S (2013) Crowdplanr: planning made easy with crowd. In: ICDE 2013, pp. 1344–1347

  29. Tian S, Zhang AX, Karger D (2020) A system for interleaving discussion and summarization in collaborative document writing. In: Conference Companion Publication of the 2020 on Computer Supported Cooperative Work and Social Computing, Association for Computing Machinery, New York, NY, USA, pp. 590–63

  30. Mesipuu M (2012) Translation crowdsourcing and user-translator motivation at facebook and skype. Transl Spaces 1(1):33–53

    Article  Google Scholar 

  31. Long TT, Huynh TD, Rosenfeld A, Ramchurn SD, Jennings NR (2015) Crowdsourcing complex workflows under budget constraints. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI’2015, AAAI Press, pp. 1298–1304

  32. Kim S, Robert LP Jr (2019) Crowdsourcing coordination: a review and research agenda for crowdsourcing coordination used for macro-tasks. Springer International Publishing, Cham, pp 17–43

    Google Scholar 

  33. Khan V, Papangelis K, Lykourentzou I, Markopoulos P (eds) (2019) Macrotask crowdsourcing - engaging the crowds to address complex problems. Human-computer interaction series. Springer, New York

    Google Scholar 

  34. Brandtner P, Auinger A, Helfert M (2014) Principles of human computer interaction in crowdsourcing to foster motivation in the context of open innovation. In: Nah FFH (ed) HCI in Business. Springer International Publishing, Cham, pp 585–596

    Chapter  Google Scholar 

  35. Fang Y, Sun H, Zhang R, Huai J, Mao Y (2014) A model for aggregating contributions of synergistic crowdsourcing workflows. In: Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, pp. 3102–3103

  36. Inasu C, Amer-YahiaSihem Chen-lei (2016) A survey of general-purpose crowdsourcing techniques. IEEE Trans Knowl Data Eng 28(9):2246–2266

    Article  Google Scholar 

  37. Fang Y, Sun H, Li G, Zhang R, Huai J (2018) Context-aware result inference in crowdsourcing. Inf Sci 460–461:346–363

    Article  Google Scholar 

  38. Huang G, Wu MH, Quinn AJ (2021) Task design for crowdsourcing complex cognitive skills. In: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–7

  39. Han T, Sun H, Song Y, Fang Y, Liu X (2021) Find truth in the hands of the few: acquiring specific knowledge with crowdsourcing. Front Comput Sci 15(4):1–12

    Article  Google Scholar 

  40. Yin B, Zeng W, Wei X (2021) Cost-effective crowdsourced join queries for entity resolution without prior knowledge. Future Gener Comput Syst

  41. Asthana S, Tobar Thommel S, Halfaker AL, Banovic N (2021) Automatically labeling low quality content on wikipedia by leveraging patterns in editing behaviors. Proc ACM Hum Comput Interact 5(CSCW2):1–23

    Article  Google Scholar 

  42. Raykar VC, Yu S, Zhao LH, Valadez GH, Florin C, Bogoni L, Moy L (2010) Learning from crowds. Mach Learn 11:1297–1322

    MathSciNet  Google Scholar 

  43. Liu Q, Peng J, Ihler AT (2012) Variational inference for crowdsourcing. In: Proceedings of the 26th Annual Conference on Neural Information Processing Systems, NIPS’2016, pp. 701–709

  44. Dawid AP, Skene AM (1979) Maximum likelihood estimation of observer error-rates using the em algorithm. Appl Stat 28:20–28

    Article  Google Scholar 

  45. Fang Y, Sun H, Chen P, Huai J (2018) On the cost complexity of crowdsourcing. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI’2018, pp. 1531–1537

  46. Whitehill J, fan Wu T, Bergsma J, Movellan JR, Ruvolo PL (2009) Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In: Y. Bengio, D. Schuurmans, J.D. Lafferty, C.K.I. Williams, A. Culotta (eds.) Advances in Neural Information Processing Systems 22, pp. 2035–2043

  47. Shani G, Pineau J, Kaplow R (2013) A survey of point-based pomdp solvers. Auton Agents Multi-Agent Syst 27(1):1–51

    Article  Google Scholar 

  48. Zhao Q, Tong L, Swami A, Chen Y (2007) Decentralized cognitive mac for opportunistic spectrum access in ad hoc networks: a pomdp framework. California Univ Davis Dept of Electrical and Computer engineering, Tech. rep

  49. Yu L, Nickerson JV (2011) Cooks or cobblers? crowd creativity through combination. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp. 1393–1402

  50. Rapp A, Cena F, Schifanella C, Boella G (2020) Finding a secure place: a map-based crowdsourcing system for people with autism. IEEE Trans Hum Mach Syst 50(5):424–433

    Article  Google Scholar 

  51. Tong Y, Zhou Z, Zeng Y, Chen L, Shahabi C (2020) Spatial crowdsourcing: a survey. VLDB J 29(1):217–250

    Article  Google Scholar 

  52. de Boer V, Hildebrand M, Aroyo L, De Leenheer P, Dijkshoorn C, Tesfa B, Schreiber G (2012) Nichesourcing: Harnessing the power of crowds of experts. In: Proceedings of the 18th International Conference on Knowledge Engineering and Knowledge Management, EKAW’2012, pp. 16–20

  53. Hosseini M, Phalp K, Taylor J, Ali R (2014) The four pillars of crowdsourcing: a reference model. In: IEEE Eighth International Conference on Research Challenges in Information Science, RCIS’2014, pp. 1–12. IEEE

  54. Hetmank L (2013) Components and functions of crowdsourcing systems-a systematic literature review. Wirtschaftsinformatik

  55. Morris RR, Dontcheva M, Gerber EM (2012) Priming for better performance in microtask crowdsourcing environments. IEEE Internet Comput 16(5):13–19

    Article  Google Scholar 

  56. Kurve A, Miller DJ, Kesidis G (2015) Multicategory crowdsourcing accounting for variable task difficulty, worker skill, and worker intention. IEEE Trans Knowl Data Eng 27:794–809

    Article  Google Scholar 

  57. Dasgupta A, Ghosh A (2013) Crowdsourced judgement elicitation with endogenous proficiency. In: Proceedings of the 22Nd International Conference on World Wide Web, WWW’2013, ACM, pp. 319–330

  58. Liu X, Lu M, Ooi BC, Shen Y, Wu S, Zhang M (2012) CDAS: a crowdsourcing data analytics system. Proc VLDB Endow 5(10):1040–1051

    Article  Google Scholar 

  59. Kovashka A, Russakovsky O, Fei-Fei L (2016) Crowdsourcing in Computer Vision. Now Publishers Inc., Hanover, MA, USA

    Book  Google Scholar 

  60. Noronha J, Hysen E, Zhang H, Gajos KZ (2011) Platemate: crowdsourcing nutritional analysis from food photographs. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST’2011, pp. 1–12

  61. Paik JH, Scholl M, Sergeev R, Randazzo S, Lakhani KR (2020) Innovation contests for high-tech procurement. Res Technol Manag 63(2):36–45

    Article  Google Scholar 

  62. Larsen-Ledet I, Korsgaard H (2019) Territorial functioning in collaborative writing. Comput Support Cooper Work (CSCW) 28(3):391–433

    Article  Google Scholar 

  63. Zaidan OF, Callison-Burch C (2011) Crowdsourcing translation: professional quality from non-professionals. In: ACL2011, ACL’2011. Association for Computational Linguistics, pp. 1220–1229

  64. Tahlil KM, Obiezu-Umeh C, Gbajabiamila T, Nwaozuru U, Oladele D, Musa AZ, Idigbe I, Okwuzu J, David AN, Bamidele TA et al (2021) A designathon to co-create community-driven hiv self-testing services for nigerian youth: findings from a participatory event. BMC Infect Dis 21(1):1–8

    Article  Google Scholar 

  65. Kolakowski M (2020) Automatic radio map creation in a fingerprinting-based ble/uwb localisation system. IET Microw Antennas Propag 14(14):1758–1765

    Article  Google Scholar 

  66. Chai C, Fan J, Li G, Wang J, Zheng Y (2019) Crowdsourcing database systems: overview and challenges. In: 35th IEEE International Conference on Data Engineering, ICDE 2019, Macao, China, April 8-11, 2019, pp. 2052–2055

  67. Ipeirotis PG, Paritosh PK (2011) Managing crowdsourced human computation: a tutorial. In: Proceedings of the 20th international conference companion on World wide web, WWW ’2011. ACM, pp. 287–288

  68. Bansal P, Eickhoff C, Hofmann T (2016) Active content-based crowdsourcing task selection. In: Proceedings of the 25th ACM International Conference on Information and Knowledge Management, CIKM’2016, pp. 529–538

  69. Karger DR, Oh S, Shah D (2014) Budget-optimal task allocation for reliable crowdsourcing systems. Oper Res 62(1):1–24

    Article  MATH  Google Scholar 

  70. Tran-Thanh L, Huynh TD, Rosenfeld A, Ramchurn SD, Jennings NR (2014) Budgetfix: budget limited crowdsourcing for interdependent task allocation with quality guarantees. In: AAMAS2014, AAMAS’2014, pp. 477–484

  71. Tran-Thanh L, Huynh TD, Rosenfeld A, Ramchurn SD, Jennings NR (2015) Crowdsourcing complex workflows under budget constraints. In: AAAI2015, AAAI’2015, pp. 1298–1304

  72. Yu H, Miao C, Shen Z, Leung C (2015) Quality and budget aware task allocation for spatial crowdsourcing. In: AAMAS2015, AAMAS’2015, pp. 1689–1690

  73. Gadiraju U, Fetahu B, Kawase R, Siehndel P, Dietze S (2017) Using worker self-assessments for competence-based pre-selection in crowdsourcing microtasks. ACM Trans Comput Hum Interact 24(4):30:1-30:26

    Article  Google Scholar 

  74. Li T (2019) Solving mysteries with the wisdom of crowds: a modularized pipeline and context slices. In: Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing, CSCW’19. Association for Computing Machinery, New York, NY, USA, pp. 68–71 https://doi.org/10.1145/3311957.3361862

  75. Law E, Zhang H (2011) Towards large-scale collaborative planning: Answering high-level search queries using human computation. In: AAAI2011, pp. 1210–1215

  76. Kittur A, Smus B, Khamkar S, Kraut RE (2011) Crowdforge: crowdsourcing complex work. In: Proceedings of the 24th annual ACM symposium on User interface software and technology, UIST’2011, ACM pp. 43–52

  77. Dai P (2013) Constructing crowdsourced workflows. Handbook of Human Computation. Springer, New York, pp 625–632

    Chapter  Google Scholar 

  78. Dai P, Lin CH, Mausam Weld DS (2013) Pomdp-based control of workflows for crowdsourcing. Artif Intell 202:52–85

    Article  MathSciNet  MATH  Google Scholar 

  79. Bernstein MS, Little G, Miller RC, Hartmann B, Ackerman MS, Karger DR, Crowell D, Panovich K (2015) Soylent: a word processor with a crowd inside. Commun ACM 58(8):85–94

    Article  Google Scholar 

  80. Feng J, Li G, Wang H, Feng J (2014) Incremental quality inference in crowdsourcing. Database Systems for Advanced Applications, DASFAA’2014. Springer, New York, pp 453–467

    Chapter  Google Scholar 

  81. Salek M, Bachrach Y, Key P (2013) Hotspotting: a probabilistic graphical model for image object localization through crowdsourcing. In: Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence, AAAI’2013. AAAI Press, pp. 1156–1162

  82. Li H, Yu B, Zhou D (2013) Error rate bounds in crowdsourcing models. arXiv preprint arXiv:1307.2674

  83. Gao C, Zhou D (2016) Minimax optimal convergence rates for estimating ground truth from crowdsourced labels. Statistics

  84. Zhang Y, Chen X, Zhou D, Jordan MI (2014) Spectral methods meet em: a provably optimal algorithm for crowdsourcing. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, NIPS’2014, pp. 1260–1268

Download references

Acknowledgements

This work was supported partly by the National Natural Science Foundation of China (No.61976187 and 92046002), and the author would thank the anonymous reviewers for helpful comments and suggestions to improve this paper.

Author information

Authors and Affiliations

  1. School of Computer and Information Engineering, Zhejiang Gongshang University, Hangzhou, 310018, China

    Yili Fang & Tao han

  2. School of Computer Science and Engineering, Beihang University, Beijing, 100191, China

    Pengpeng Chen

Authors
  1. Yili Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pengpeng Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tao han
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yili Fang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (pdf 125 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fang, Y., Chen, P. & han, T. Hint: harnessing the wisdom of crowds for handling multi-phase tasks. Neural Comput & Applic 35, 22911–22933 (2023). https://doi.org/10.1007/s00521-021-06825-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06825-7

Keywords

Search

Navigation