research-article

Classification and evaluation of IoT brokers: : A methodology

Authors:

Eddas Bertrand‐Martinez,

Phelipe Dias Feio,

Vagner de Brito Nascimento,

Antônio AbelémAuthors Info & Claims

International Journal of Network Management, Volume 31, Issue 3

https://doi.org/10.1002/nem.2115

Published: 04 May 2021 Publication History

Summary

Since the term Internet of Things (IoT) was coined by Kevin Ashton in 1999, a number of middleware platforms have been developed to cope with important challenges such as the integration of different technologies. In this context of heterogeneous technologies, IoT message brokers become critical elements for the proper function of smart systems and wireless sensor networks (WSN) infrastructures. There are several evaluations made on IoT messaging middleware performance. Nevertheless, most of them ignore crucial aspects of the IoT context that also need to be included, such as reliability and other qualitative aspects. Thus, in this article, we propose a methodology for classification and evaluation of IoT brokers to help the scientific community and technology industry on evaluating them according to their interests, without leaving out important aspects for the context of smart environments. Our methodology bases its qualitative evaluations on the ISO/IEC 25000 (SQuaRE) set of standards and its quantitative evaluations on Jain's process for performance evaluation. We developed a case study to illustrate our proposal with 12 different open‐source brokers, validating the feasibility of our methodological approach.

Graphical Abstract

This work defines a methodology for software quality evaluation of message brokers for Internet of Things. It provides also a theoretical framework for scientists to develop their own metrics system to evaluate quality in IoT middleware, and showcases a practical implementation of the methodology in order to make it more didactic.

References

[1]

Chen S, Xu H, Liu D, Hu B, Wang H. A vision of IoT: Applications, challenges, and opportunities with China perspective. IEEE Internet of Things Journal. 2014;1(4):349‐359.

[2]

Ma R, Lam P, Leung C. Potential pitfalls of smart city development: A study on parking mobile applications (apps) in Hong Kong. Telematics and Informatics. 2018;35(6):1580‐1592.

[3]

Ellsmoor J. Smart Cities: The Future Of Urban Development. https://www.forbes.com/sites/jamesellsmoor/2019/05/19/smart-cities-the-future-of-urban-development/\#7684ffa12f90; 2019.

[4]

Caragliu A, Del Bo C, Nijkamp P. Smart cities in Europe. Journal of Urban Technology. 2011;18(2):65‐82.

[5]

Del Esposte AM, Kon F, Costa FM, Lago N. InterSCity: a scalable microservice‐based open source platform for smart cities. In Proceedings of the 6th International Conference on Smart Cities and Green ICT Systems (SMARTGREENS 2017). SCITEPRESS ‐ Science and Technology Publications, Lda, Setubal, PRT, 2017: 35‐46. https://doi.org/10.5220/0006306200350046

[6]

Nascimento N, Nascimento C. FIoT: An agent‐based framework for self‐adaptive and self‐organizing applications based on the Internet of Things. Information Sciences. 2017;378:161‐176.

[7]

Batista C, Silva PV, Cavalcante E, et al. A middleware environment for developing internet of things applications. In Proceedings of the 5th Workshop on Middleware and Applications for the Internet of Things (M4IoT'18). New York, NY, USA: Association for Computing Machinery; 2018:41‐46. https://doi.org/10.1145/3286719.3286728

Digital Library

[8]

Unicamp . Smart Campus ‐ Unicamp. https://smartcampus.prefeitura.unicamp.br/ (Portuguese).

[9]

Naik N. Choice of effective messaging protocols for IoT systems: MQTT, CoAP, AMQP and HTTP. Vienna: IEEE International Systems Engineering Symposium (ISSE); 2017:1‐7. https://doi.org/10.1109/SysEng.2017.8088251

[10]

Dizdarevic J, Carpio F, Jukan A, Masip‐Bruin X. A survey of communication protocols for Internet of Things and related challenges of fog and cloud computing integration. ACM Comput Sur. 2019;51:1‐29.

Digital Library

[11]

Kraijak S, Tuwanut P. A survey on IoT architectures, protocols, applications, security, privacy, real‐world implementation and future trends. 11th International Conference on Wireless Communications, Networking and Mobile Computing (WiCOM 2015). Shanghai; 2015:1‐6. https://doi.org/10.1049/cp.2015.0714

[12]

ISO/IEC . Systems and software engineering – systems and software quality requirements and evaluation (SQuaRE) – guide to SQuaRE. International Standard ISO/IEC 25000:2014; 2014.

[13]

Raj J. The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation and Modeling. New York, USA: Wiley‐ Interscience; 1991.

[14]

Sethi P, Sarangi S. Internet of Things: Architectures, protocols and applications. J Elect Comp Eng. 2017;2017:1‐25.

Digital Library

[15]

Razzaque M, Milojevic‐Jevric M, Palade A, Clarke S. Middleware for Internet of Things: A survey. IEEE Int Things J. 2016;3:70‐95.

[16]

Bhawiyuga A, Kartikasari DP, Pramukantoro ES. A publish subscribe based middleware for enabling real time web access on constrained device. 2017 9th International Conference on Information Technology and Electrical Engineering (ICITEE). Phuket, Thailand; 2017:1‐5. https://doi.org/10.1109/ICITEED.2017.8250510

[17]

Giambona R, Redondi AEC, Cesana M. Demonstrating MQTT+: an advanced broker for data filtering, processing and aggregation. In Proceedings of the 21st ACM International Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems (MSWIM'18). New York, NY, USA: Association for Computing Machinery; 2018:357‐358. https://doi.org/10.1145/3242102.3243317

[18]

Pipatsakulroj W, Visoottiviseth V, Takano R. muMQ: a lightweight and scalable MQTT broker. 2017 IEEE International Symposium on Local and Metropolitan Area Networks (LANMAN). Osaka, Japan; 2017:1‐6. https://doi.org/10.1109/LANMAN.2017.7972165

[19]

Blackstock M, Kaviani N, Lea R, Friday A. MAGIC broker 2: an open and extensible platform for the internet of things. 2010 Internet of Things (IOT). Tokyo, Japan; 2010:1‐8. https://doi.org/10.1109/IOT.2010.5678443

[20]

Antonic A, Roankovic K, Marjanovic M, Pripuic K, arko IP. A mobile crowdsensing ecosystem enabled by a cloud‐based publish/subscribe middleware. 2014 International Conference on Future Internet of Things and Cloud, Barcelona. Barcelona, Spain; 2014:107‐114. https://doi.org/10.1109/FiCloud.2014.27

Digital Library

[21]

Vandikas K, Tsiatsis V. Performance Evaluation of an IoT Platform. 2014 Eighth International Conference on Next Generation Mobile Apps, Services and Technologies. Oxford, United Kingdom; 2014: 141‐146. https://doi.org/10.1109/NGMAST.2014.66

Digital Library

[22]

Banno R, Sun J, Fujita M, Takeuchi S, Shudo K. Dissemination of edge‐heavy data on heterogeneous MQTT brokers. 2017 IEEE 6th International Conference on Cloud Networking (CloudNet). Prague; 2017:1‐7. https://doi.org/10.1109/CloudNet.2017.8071523

[23]

Jutadhamakorn P, Pillavas T, Visoottiviseth V, Takano R, Haga J, Kobayashi D. A scalable and low‐cost MQTT broker clustering system. 2017 2nd International Conference on Information Technology (INCIT). Nakhon Pathom, Thailand; 2017:1‐5. https://doi.org/10.1109/INCIT.2017.8257870

[24]

Neumann MA, Bach CT, Miclaus A, Riedel T, Beigl M. Always‐On web of things infrastructure using dynamic software updating. In Proceedings of the Seventh International Workshop on the Web of Things (WoT'16). New York, NY, USA: Association for Computing Machinery; 2016:5‐10. https://doi.org/10.1145/3017995.3017997

Digital Library

[25]

Mishra B. Performance evaluation of MQTT broker servers. In: Gervasi O. et al., ed. Computational Science and Its Applications – ICCSA 2018. Lecture Notes in Computer Science, vol. 10963. Springer, Cham; 2018. https://doi.org/10.1007/978-3-319-95171-3_47

[26]

Scalagent . Benchmark of MQTT servers; 2015.

[27]

Ismail AA, Hamza HS, Kotb AM. Performance evaluation of open source IoT platforms. 2018 IEEE Global Conference on Internet of Things (GCIoT). Alexandria, Egypt; 2018:1‐5. https://doi.org/10.1109/GCIoT.2018.8620130

[28]

Daniel H, Niels K, Thomas M, Vlado H, Adam W. Meeting IoT platform requirements with open pub/sub solutions. Annal Tel. 2017;72:41‐52.

[29]

Pereira C, Cardoso J, Aguiar A, Morla R. Benchmarking pub/sub IoT middleware platforms for smart services. J Reliable Int Env. 2018;4:25‐37.

[30]

European Comission . The Lighthouse Project IoT‐A. https://cordis.europa.eu/project/id/257521, FP7‐ICT ‐ Specific Programme “Cooperation”: Information and communication technologies; 2013.

[31]

Glinz M. Software Product Quality. https://www.ifi.uzh.ch/dam/jcr:8e9cd2bb-60de-4ce7-b17e-8f994744d4cd/06_product_q.pd; 2014.

[32]

Boehm BW, Brown JR, Lipow M. Quantitative evaluation of software quality. In Proceedings of the 2nd international conference on Software engineering (ICSE'76). Washington, DC, USA: IEEE Computer Society Press; 1976:592‐605.

Digital Library

[33]

Brunst H, Mueller M. Performance Analysis of Computer Systems: Requirements, metrics, techniques, and mistakes. https://tu-dresden.de/zih/ressourcen/dateien/lehre/ws1112/lars/vorlesungen/lars_lecture_02_requirements-metrics-techniques.pdf. 2019.

[34]

ISO/IEC. Systems and software engineering – systems and software quality requirements and evaluation (SQuaRE) – system and software quality models. International Standard ISO/IEC 25010:2011; 2011.

[35]

Schneider F, Berenbach F. A literature survey on international standards for systems requirements engineering. Procedia Computer Science. Vol. 16, Atlanta, USA; 2013:796‐805. https://doi.org/10.1016/j.procs.2013.01.083

[36]

McCall J, Matsumoto M. Software Quality Measurement Manual. Rome Air Development Center, RADC‐TR‐80‐109‐Vol‐2; 1980.

[37]

Deissenboeck F, Heinemann L, Herrmannsdoerfer M, Lochmann K, Wagner S. The quamoco tool chain for quality modeling and assessment. 2011 33rd International Conference on Software Engineering (ICSE). Honolulu, HI; 2011:1007‐1009. https://doi.org/10.1145/1985793.1985977

[38]

ISO/IEC . Systems and software engineering ‐ systems and software quality requirements and evaluation (SQuaRE) ‐ evaluation process. International Standard ISO/IEC 25040:2011; 2011.

[39]

Eclipse Foundation . Mosquitto ‐ An open source MQTT broker. https://mosquitto.org/

[40]

Apache Foundation . ActiveMQ ‐ Flexible & Powerful Open Source Multi‐Protocol Messaging. https://activemq.apache.org/

[41]

Apache Foundation . The Apollo Project. https://github.com/apache/activemq-apollo

[42]

Pivotal Software . RabbitMQ ‐ Messaging that just works. https://www.rabbitmq.com/

[43]

Eclipse Foundation . Moquette MQTT Broker. https://moquette-io.github.io/moquette/

[44]

Lee F. The EMQ project ‐ The Massively Scalable MQTT Broker for IoT and Mobile Applications. https://emqtt.io

[45]

Christian G. HiveMQ Open Source Community. https://www.hivemq.com/developers/community/

[46]

Nicolas J. HBMQTT. https://github.com/beerfactory/hbmqtt

[47]

Matteo C. Mosca ‐ MQTT broker as a module. https://www.mosca.io/

[48]

Emitter Studios . Emitter ‐ Scalable Real‐Time Communication Accross Devices. https://emitter.io/

[49]

GRID System . GRID Server Open Sourced. https://grids.systems/2019/08/02/grid-server-open-sourced

[50]

Banks A, Gupta R. MQTT Version 3.1.1. https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html

[51]

Banks A, Briggs E, Borgendale K, Gupta R. MQTT Version 5.0. https://docs.oasis-open.org/mqtt/mqtt/v5.0/cs02/mqtt-v5.0-cs02.html

[52]

Bertrand‐Martinez E, Feio P, Nascimento V, Pinheiro B, Abelém A. A methodology for classification and evaluation of IoT brokers. 9th Latin American Network Operations and Management Symposium (LANOMS). Niteroi, Brazil; 2019. ISSN: 978‐3‐903176‐23‐2. http://dl.ifip.org/db/conf/lanoms/lanoms2019/195912_1.pdf

Cited By

Freitas NRocha AM-Oliveira FAlemão DBarata J(2024)Reconfigurable Framework for Data Extraction Using Interoperable Brokers in ManufacturingSN Computer Science10.1007/s42979-024-03124-55:7Online publication date: 16-Aug-2024
https://dl.acm.org/doi/10.1007/s42979-024-03124-5
Papadakis NBouloukakis GMagoutis KKemme BRiviere ESchiavoni VPasin M(2023)ComDeX: A Context-aware Federated Platform for IoT-enhanced CommunitiesProceedings of the 17th ACM International Conference on Distributed and Event-based Systems10.1145/3583678.3596890(37-48)Online publication date: 27-Jun-2023
https://dl.acm.org/doi/10.1145/3583678.3596890
Sochor HFerrarotti FRamler R(2023)An automated evaluation of broker compatibility for the Message Queuing Telemetry Transport protocolJournal of Software: Evolution and Process10.1002/smr.241035:7Online publication date: 2-Jul-2023
https://dl.acm.org/doi/10.1002/smr.2410

Index Terms

Classification and evaluation of IoT brokers: A methodology

Index terms have been assigned to the content through auto-classification.

Recommendations

Performance evaluation of IoT middleware

Connected objects in a billion scale are expected in the Internet of Things (IoT). These objects (things) are interacting autonomously, with minimal human intervention and connected to the Internet. Most of these objects are constrained in terms of ...
Authentication mechanisms for IoT system based on distributed MQTT brokers: review and challenges
Abstract
With the rapid growth of internet connected devices and the tremendous amount of data that could be generated and exchanged in IoT environment, we need to reconsider in the current IoT architecture that based on Cloud computing system, to avoid ...
Intelligent Brokers in an Internet of Things for Logistics
ICC '16: Proceedings of the International Conference on Internet of things and Cloud Computing

The proliferation of Internet of Things (IoT) logistics applications necessitates the availability of new generation 'brokers' to bridge the smart objects to each other and to the enterprise world (IT systems). Such brokers need to be able to cope with ...

Comments

Information & Contributors

Information

Published In

© 2020 John Wiley & Sons, Ltd.

Publisher

John Wiley & Sons, Inc.

United States

Publication History

Published: 04 May 2021

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Freitas NRocha AM-Oliveira FAlemão DBarata J(2024)Reconfigurable Framework for Data Extraction Using Interoperable Brokers in ManufacturingSN Computer Science10.1007/s42979-024-03124-55:7Online publication date: 16-Aug-2024
https://dl.acm.org/doi/10.1007/s42979-024-03124-5
Papadakis NBouloukakis GMagoutis KKemme BRiviere ESchiavoni VPasin M(2023)ComDeX: A Context-aware Federated Platform for IoT-enhanced CommunitiesProceedings of the 17th ACM International Conference on Distributed and Event-based Systems10.1145/3583678.3596890(37-48)Online publication date: 27-Jun-2023
https://dl.acm.org/doi/10.1145/3583678.3596890
Sochor HFerrarotti FRamler R(2023)An automated evaluation of broker compatibility for the Message Queuing Telemetry Transport protocolJournal of Software: Evolution and Process10.1002/smr.241035:7Online publication date: 2-Jul-2023
https://dl.acm.org/doi/10.1002/smr.2410

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents