Abstract
MapReduce is a design pattern for processing large datasets distributed on a cluster. Its performances are linked to the data structure and the runtime environment. Indeed, data skew can yield an unfair task allocation, but even when the initial allocation produced by the partition function is well balanced, an unfair allocation can occur during the reduce phase due to the heterogeneous performance of nodes. For these reasons, we propose an adaptive multi-agent system. In our approach, the reducer agents interact during the job and the task reallocation is based on negotiation in order to decrease the workload of the most loaded reducer and so the runtime. In this paper, we propose and evaluate two negotiation strategies. Finally, we experiment our multi-agent system with real-world datasets over heterogeneous runtime environment.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Find the latest articles, discoveries, and news in related topics.Notes
This paper is an extension of (Baert et al. [1]). We provide here a deeper description of the framework including some illustrative examples. Finally, we present/evaluate some additional strategies and experiments measuring the speedup of our MAS in a distributed environment.
It is worth noticing that the task delegation is conservative.
References
Baert Q, Caron A-C, Morge M, Routier J-C (2016) Fair multi-agent task allocation for large data sets analysis, In: Proceedings of 14th international conference on practical applications of agents and multi-agent systems (PAAMS), vol 9662, Sevilla, Spain, LNAI, Springer, pp 24–35
Brandt F, Conitzer V, Endriss U, Lang J, Procaccia A D (2016) Handbook of computational social choice. Cambridge University Press, Cambridge
Clair G, Kaddoum E, Gleizes MP, Picard G (2008) Self-regulation in self-organising multi-agent systems for adaptive and intelligent manufacturing control, In: Proceedings of the second IEEE international conference on self-adaptive and self-organizing systems, SASO 2008, 20–24 Oct 2008, Venice, Italy, pp 107–116
Clinger WD (1981) Foundations of actor semantics. PhD thesis, Massachusetts Institute of Technology
Dean J, Ghemawat S (2008) Mapreduce: simplified data processing on large clusters. Commun ACM 51(1):107–113
DeCandia G, Hastorun D, Jampani M, Kakulapati G, Lakshman A, Pilchin A, Sivasubramanian S, Vosshall P, Vogels W (2007) Dynamo: Amazon’s highly available key-value store. In: Proceedings of the 21st ACM SIGOPS symposium on operating systems principles (SOSP ’07), pp 205–220
Essa YM, Attiya G, El-Sayed A (2014) Mobile agent based new framework for improving big data analysis. Int J Adv Comput Sci Appl 5(3):25–32
Hewitt C (1977) Viewing control structures as patterns of passing messages. Artif Intell 8(3):323–364
Kwon Y, Balazinska M, Howe B, Rolia J (2012a) Skewtune: mitigating skew in mapreduce applications. In: Proceedings of the 2012 ACM SIGMOD international conference on management of data, SIGMOD’12’, pp 25–36
Kwon Y, Balazinska M, Howe B, Rolia JA (2012b) Skewtune in action: mitigating skew in mapreduce applications. PVLDB 5(12):1934–1937
Lama P, Zhou X (2012) Aroma: automated resource allocation and configuration of mapreduce environment in the cloud. In: Proceedings of the 9th internatinal conference on autonomic computing (ICAC’12), pp 63–72
Liroz-Gistau M, Akbarinia R, Agrawal D, Valduriez P (2016) FP-hadoop: efficient processing of skewed mapreduce jobs. Inf Syst 60:69–84
Morge M, Stathis K, Vercouter L (2009) Arguing over motivations within the V3A-architecture for self-adaptation. In: Proceedings of the international conference on agents and artificial intelligence (ICAART), Porto, Portugal, 19–21 Jan 2009, pp 214–219
Nongaillard A, Mathieu P (2011) Egalitarian negotiations in agent societies. Appl Artif Intell 25(9):799–821
Odell J, Parunak HVD, Bauer B (2000) Extending UML for agents. In: Wagner G, Lesperance Y, Yu E (eds) Proceedings of the agent-oriented information systems workshop at the 17th national conference on artificial intelligence
Rihawi O, Secq Y, Mathieu P (2015) Load-balancing for large scale situated agent-based simulations. In: Proceedings of the international conference on computational science, ICCS 2015, computational science at the gates of nature, Reykjavík, Iceland, 1–3 June 2015, 2014, pp 90–99
Serugendo GDM, Gleizes M-P, Karageorgos A (2005) Self-organization in multi-agent systems. Knowl Eng Rev 20(02):165–189
Slagter K, Hsu C, Chung Y, Zhang D (2013) An improved partitioning mechanism for optimizing massive data analysis using mapreduce. J Supercomput 66(1):539–555
Smith R (1980) The contract net protocol: highlevel communication and control in a distributed problem solver. IEEE Trans Comput C 29:12
Wagner IA, Lindenbaum M, Bruckstein AM (1999) Distributed covering by ant-robots using evaporating traces. IEEE Trans Robot Autom 15(5):918–933
Wolf JL, Balmin A, Rajan D, Hildrum K, Khandekar R, Parekh S, Wu K, Vernica R (2012) On the optimization of schedules for mapreduce workloads in the presence of shared scans. VLDB J 21(5):589–609
Zaharia M, Xin RS, Wendell P, Das T, Armbrust M, Dave A, Meng X, Rosen J, Venkataraman S, Franklin MJ, Ghodsi A, Gonzalez J, Shenker S, Stoica I (2016) Apache spark: a unified engine for big data processing. Commun ACM 59(11):56–65
Acknowledgements
This project is supported by the CNRS Challenge Mastodons. The authors would like to thank the scientific committee of the International Conference on Practical Applications on Multi-agent Systems for the invitation in this special issue and the reviewers for their useful suggestions.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Baert, Q., Caron, AC., Morge, M. et al. Fair multi-agent task allocation for large datasets analysis. Knowl Inf Syst 54, 591–615 (2018). https://doi.org/10.1007/s10115-017-1087-4
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10115-017-1087-4