Volume 6, Issue 5, May – 2021 International Journal of Innovative Science and Research Technology

ISSN No:-2456-2165

Comparison of IoT Protocols Performance

Yaseen NADIR SUPERVISOR

Information Institution Dr. Mutlu Tahsin ÜSTÜNDAĞ
Gazi University Assoc. Prof., Gazi Faculty of Education
Ankara, Turkey Gazi University
Ankara, Turkey

Abstract:- Internet of Things, it is been more than a and MQTT in terms of traffic, packet loss probability, and
decade since this concept was introduced to the society. latency conditions. Lavinia Nastasa in her research
In this research we aim to compare three IoT application concentrates on application layer protocols; CoAP, MQTT,
protocols; Advanced Messaging Queuing Protocol and XMPP from security point of view, described briefly the
(AMQP), Message Queuing Telemetry Transport three protocols and their vulnerabilities, and according to the
(MQTT) and Constrained Application Protocol (CoAP). study, it is none of them are best for any type of solutions in
The selected protocols efficiency will be evaluated using terms of both security and functionality (Nastasa, 2017).
indicators related to Throughput and Round-Trip time Alvin VALERA and Hwee TAN, have proposed a common
(RTT). In the considered scenario an IoT device will middleware using common application programming
transfer data to a server and waits for the response. The interface, and have tested CoAP and MQTT protocols, the
data will be sent in different sizes of packets. We have research results shows that; different network conditions can
proposed a testbed using python programing language’s affects the performance of different protocols, MQTT has
library SciPy and socket programming to evaluate the lower delay for lower packet loss and vice versa, CoAP
three protocols. Experimentation tests reveal which generates less traffic to ensure reliable delivery when the
protocol is best suited for different scenarios accordingly. massage size is small and loss rate is equal or less than 25%
Results show that overall MQTT achieves the highest (Valera & Tan, 2014). In (Paolo Bellavista, Alessandro
protocols efficiency among other protocols. Zanni., 2016), the authors proposed an innovative scalable
distributed architecture for efficient IoT-cloud integration.
Keywords:- IoT, IoT Protocols, Performance, MQTT, The two selected protocols were CoAP and MQTT. The
AMQP, COAP. performance benchmark for the time required to complete
the transmission of 1000, 10000 and 60000 messages was
I. INTRODUCTION set and according to the authors, the MQTT protocol was
much faster than CoAP.
The internet is the most important and transformative
technology have ever invented, it is like digital fabric that There are several published survey papers regard IoT
woven into our lives and rapidly changed the world. application layer protocols but as IoT technology is growing
Likewise, new technology has emerged and it is poised to tremendously fast, it was important to test these application
change the world again, this technology is not about protocols with different metrics. This experiment will
connecting people or computers, it is about connecting examine the following selected application services used for
objects “The Internet of Things”. The internet of things IoT data transfer:
represents a vision wherein the net extends into the actual
world embracing daily objects. Physical objects are no A. Constrained Application Protocol (CoAP)
longer disconnected from the digital world, however these Constrained Application Protocol was created and
objects can be managed remotely and can act as physical standardized by the Internet Engineering Task Force (IETF)
access points to Internet services. In the Network of things, Constrained Restful Environment working group (CoRE).
objects can start to share experience with other objects, by CoAP is specialized internet application protocol for
adding the ability to sense, communicate to control and constrained nodes and networks based on Representational
collaborate with each other, things that are connecting to the State Transfer (REST) (Shelby, Z; Hartke, K; Bormann,
internet allowing them to generate, send, receive and 2014). CoAP is a RESTful application protocol but in order
exchange data. to understand CoAP clearly the REST concept must be
introduced first. Representational State Transfer (REST) is a
IoT communication protocols are a subject of many web architectural style. RESTful systems are classified as
studies in research community. Bhattacharyya et al. (Soma they are stateless and separate concerns of client and servers.
Bandyopadhyay, Abhijan Bhattacharyya., 2013) have done a REST architecture is request-response architecture, client
comparison between CoAP's request-response model and sends request to server in order to get to stored resources in
MQTT publish-subscribe model. Application layer rules for it and these each resource has a Uniform Resource Identifier
IoT are explored in (Sotirios Kontogiannis, Angelos (URI) as an address. In case where server response, a
Chatzimparmpas, George Kokkonis., 2015); here, the content-type must be included in the header of the response.
authors have presented quantitative examinations of CoAP Unlike REST, CoAP uses datagram-oriented transport such

IJISRT21MAY311 www.ijisrt.com 556

Volume 6, Issue 5, May – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
as UDP layer to keep design simple, with a primary goal of is done according to its design. Since the MQTT protocol is
providing communication that works between devices with based on TCP/IP and it is implementations responsibility to
limited resources or networks with low bandwidth. CoAP provide appropriate security and integrity, security features
supports the use of multicast IP destination addresses, must be implemented on top of MQTT. This can be
enabling multicast CoAP requests. The CoAP message achieved by using Transport Layer Security TLS/Secure
format is encoded in Binary format (Shelby, Z; Hartke, K; Socket Layer SSL. Authentication is based on certificates
Bormann, 2014). CoAP message format starts with four and privacy is based on encryption mechanism by the
fixed-size 4 bytes header consisting five fields. As CoAP application.
runs over UDP, it is secured using Datagram Transport Objectives of the Study;
Layer Security. - Evaluating and comparing the performance of IoT
protocols for different network scenarios.
B. Advance Messaging Queuing Protocol (AMQP) - Is protocol efficiency affected by different conditions of
The Advance Messaging Queuing Protocol is a binary the network?
and an open standard application layer protocol for message - Is the Round Trip Time affected by network conditions?
oriented middleware. It is been designed to provide peer-to-
peer (point-to-point and publish and subscribe) routing, Although there are many studies on the Internet of
message orienting, queuing, reliability based on underlying Things, It is believed that there are still a need for
on transport layer protocol such as Transmission Control performance analysis studies in different conditions of
Protocol (TCP), security based on Simple Authentication different networks. Given the significant impact of this
and Security Layer (SASL) and Transport Layer Security situation, more studies are needed. In this study, evaluation
(TLS). The Advanced Message Queuing Protocol (AMQP) of the effectiveness of IOT protocols and comparison will be
was released by various financial institutes and software examined. Information about the selected protocols will be
companies in 2006 and was standardized by OASIS1 in the collected based on the previous studies, these protocols will
year 2012 (Turowski, Bosse, Kubela, & Pohl, 2018). be evaluated and analyzed with the proposed testbed and
comparisons will be revealed within the framework of the
AMQP’s main components are clients and servers determined indicators.
(brokers). Broker consist of: exchanges, message queue and
binding, message routing occur using these three The methodology of obtaining, testing and analyzing
components. Exchange receives messages from publisher data is based on several stages:
and rout it to the appropriate queue. Message Queue stores Firstly: Learning and studying IoT protocols MQTT,
the messages until the meant consumer client (subscriber) AMQP, COAP and these studies will be used for scientific
processes it safely. Biding, defines the relationships between research.
exchanges and queues. There are two security approaches Secondly: Designing the proposed testbed with different
provided by AMQP: Transport layer Security (TLS), and scenarios using python programming language.
Simple Authentication and Security Layer SASL. Thirdly: implementing, simulating, and testing IoT
protocols.
C. Message Queue Telemetry Transport (MQTT) Fourthly: After the implementation and testing of the
MQTT is message-oriented protocol, a client server protocol, the data will be analyzed using Anaconda
publish-subscribe message transport protocol. It is an environment to facilitate the qualitative analysis process.
extremely light-weight, open, simple, designed to be easy to
implement for high-latency or unreliable network II. EXPERIMENTAL DESIGN
(Konstantinos, et al., 2016).The MQTT protocol run over
TCP/IP networks. MQTT was created by IBM then In this section, we discuss the implementation of the
standardized by OASIS. The publish-subscribe model of proposed testbed and evaluate the performance of its
MQTT is based on the Client-Server model, and the server is proposed algorithm with different routing protocols, e.g.
more like a broker or gateway, an MQTT Broker is a device AMQP, MQTT, and COAP.
that acts as intermediary between client which publish and
client which has made subscription. MQTT delivers The proposed software consists of three structures.
application messages with three QoS levels: They are the IoT sensor structure, the gateway module
 QoS 0: “At most once”, where the message is delivered structures, and the cloud application server structures. The
but no response is sent by the receiver and no resend first structure is the IoT sensor, where the proposed software
message is performed by the sender, lost can occur. simulates a light version of the IoT device, which generates
 QoS 1: “At least once”, this QoS enures the delivery of packets in three different packet formats represent the
message to the receiver at least once, duplicated can routing protocols, i.e., COAP, MQTT, and AMQP. The
occur. second structure is the gateway module, which is up and
 QoS 2: “Exactly once delivery”, this is where the waiting to sniff different packets coming from a group of
message is assured to arrive at the receiver with no loss IoT sensors and forward them to the cloud application server
or duplication, it is the highest level of QoS. acting as the man in the middle. The third structure is the
cloud application server, which is up and running tending on
The important part is that there is no security features its listening port to receive connection requests from the
in MQTT specifications, the security of each implementation

IJISRT21MAY311 www.ijisrt.com 557

Volume 6, Issue 5, May – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
gateway. This server represents the Application Enablement Figure 2 depicts the upstream and downstream
Platform (AEP) in the IoT value chain. throughput of the gateway in the case of AMQP routing
protocol. It shows that downstream throughput is almost four
In our simulation, the performance evaluation of the times higher than upstream throughput. Also, as the total
proposed algorithm has been achieved taking into number of packets sent by IoT devices increases, the
consideration the following evaluation metrics: upstream and downstream throughput of the gateway
• Upstream processing delay: it is the execution time to decreases slightly.
decapsulate the IoT device packet header, encapsulate the
payload into the TCP/UDP header and send the packet to the
cloud application server.
• Downstream processing delay: it is the execution time to
decapsulate the TCP/UDP header, encapsulate the payload
into the IoT device packet header and send the reply back to
the corresponding IoT device.
• Average processing delay: it is the round-trip processing
time to process the IoT device packet while traveling in both
upstream and downstream directions.
• Upstream throughput: it is the total number of packets per
second that the gateway can handle in the upstream direction
from the IoT device to the cloud application server.
• Downstream throughput: it is the total number of packets
per second that the gateway can handle in the downstream
direction from the cloud application server to the IoT device. Figure 2: The relation between throughput and number of
connected devices in AMQP.
III. RESULTS AND DISCUSSION
E. Message Queuing Telemetry Transport (MQTT)
We have tested the performance by changing the Figure 3 shows that in the MQTT protocol, the average
number of packets sent by IoT devices, and check the upstream processing delay is shorter than the average
average processing delay of the upstream, the downstream, downstream processing delay. It means that the packet
and the round-trip processing delay, besides the upstream decapsulation from MQTT packets and encapsulation into
and downstream throughput. the TCP/UDP header takes lower processing time than
decapsulating the payload from the TCP/UDP header and
D. Advanced Messaging Queuing Protocol (AMQP) encapsulating it into the MQTT header.
Figure 1 shows that in AMQP protocol, the average
upstream processing delay is shorter than the average Besides, as the total number of connected devices
downstream processing delay. It means that the packet increases, the upstream processing delay is almost constant
decapsulation from AMQP packets and encapsulation into for different numbers of connected devices, while the
the TCP/UDP header takes lower processing time than downstream processing time increases, accordingly the total
decapsulating the payload from the TCP/UDP header and average round-trip processing delay increases as well. This
encapsulating it into the AMQP header. Moreover, increasing means the total number of connected devices and the
the total number of packets sent from the IoT devices doesn’t downstream processing time have a direct effect on the
have a high effect on the average upstream processing delay, scalability performance.
while it has a high effect on the downstream processing
delay, accordingly downstream processing delay has a direct
effect on the total round-trip processing time.

Figure 1: The relation between average processing delay and Figure 3: The relation between average processing delay and
number of connected devices in AMQP. number of connected devices in MQTT.

IJISRT21MAY311 www.ijisrt.com 558

Volume 6, Issue 5, May – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
Figure 4 depicts the upstream and downstream Figure 6 shows the upstream and downstream
throughput of the gateway in the case of the MQTT routing throughput of the gateway in the case of COAP routing
protocol. It shows that downstream throughput is almost protocol. It shows that downstream throughput is almost six
seven times higher than upstream throughput. Also, as the times higher than upstream throughput. Also, as the total
total number of packets sent by IoT devices increases, the number of packets sent by IoT devices increases, the
downstream throughput of the gateway decreases slightly. downstream throughput of the gateway decreases.

Figure 4: The relation between throughput and number of Figure 6: The relation between throughput and number of
connected devices in MQTT. connected devices in COAP.

F. Constrained Application Protocol (COAP) Figure 7 shows the relation between the average
Figure 5 shows that in COAP protocol, the average processing delay and the number of connected IoT devices
upstream processing delay is shorter than the average for AMQP, MQTT, and COAP routing protocols. It is
downstream processing delay. It means that the packet obvious that for small traffic volume, e.g., 150 connected
decapsulation from COAP packets and encapsulation into devices, AMQP routing protocol has the highest average
the TCP/UDP header takes lower processing time than round-trip processing delay among other routing protocols.
decapsulating the payload from the TCP/UDP header and COAP routing protocol comes second, and then MQTT has
encapsulating it into COAP header. the lowest average round-trip processing delay. While for
the high traffic volume, e.g., 550 connected devices and
Also, as the total number of connected devices beyond, AMQP routing protocol provides lower round-trip
increases, the upstream processing delay is almost constant processing delay than COAP, however MQTT still the
for different numbers of connected devices, while the lowest round-trip processing delay among other routing
downstream processing time increases, accordingly the total protocols.
average round-trip processing delay increases as well. This
means the total number of connected devices and the
downstream processing time have a direct effect on the
average round-trip processing delay.

Figure 7: The relation between average processing delay and

number of connected devices for all routing

On the other hand, Figure 8 depicts the relation

Figure 5: The relation between average processing delay and between the upstream and downstream throughput with the
number of connected devices in COAP. total number of connected IoT devices for AMQP, MQTT,
and COAP routing protocols. It is obvious that there is no

IJISRT21MAY311 www.ijisrt.com 559

Volume 6, Issue 5, May – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
high difference in the upstream throughput for all routing REFERENCES
protocols, as they are almost closed to each other. While in
the downstream throughput, the results show that the MQTT [1]. Konstantinos, Fysarakis, Ioannis Askoxylakis, Othonas
routing protocol has the highest throughput, and COAP is Soultatos, Ioannis Papaefstathiou, Charalampos
the second highest, then AMQP is the lowest throughput Manifavas, and Vasilios Katos. 2016. "Which IoT
among all routing protocols. Protocol? Comparing standardized approaches over a
common M2M application."
[2]. Nastasa, Lavinia. 2017. "Security in the Interner of
Things: A survey on Application Layer Protocol."
[3]. Paolo Bellavista, Alessandro Zanni. 2016. "Towards
Better Scalability for IoT-Cloud Interactions." IEEE
2nd International Forum on Research and
Technologies for Society and Industry Leveraging a
better tomorrow (RTSI).
[4]. Shelby, Z; Hartke, K; Bormann. 2014. "The
Constrained Application Protocol (CoAP)." RFC 7252.
[5]. Soma Bandyopadhyay, Abhijan Bhattacharyya. 2013.
"Lightweight Internet Protocols for Web Enablement
of Sensors using Constrained." International
Conference on Computing, Networking and
Communications, Workshops Cyber Physical System.
[6]. Sotirios Kontogiannis, Angelos Chatzimparmpas,
Figure 8: The relation between throughput and number of George Kokkonis. 2015. "Middleware IoT protocols
connected devices for all routing protocols. performance evaluation for carrying out clustered
data."
IV. CONCLUSION [7]. Turowski, Klaus, Sascha Bosse, Janick Kubela, and
Matthias Pohl. 2018. "Performance Evaluation of
In this research the selected application layer protocols Application Layer."
of Internet of things are been simulated using the proposed [8]. Valera, Alvin Cerdeva, and Hwee Xian Tan. 2014.
testbed software and number of tests with different volume "Performance Evaluation of MQTT and CoAP.
traffics were run on the three protocols performance.

The results show that for low volume round-trip time,

every protocol has almost the same round-trip processing
delay. While the number of packets increases the difference
between the protocols increases too, as a result for high
volume round-trip time, MQTT has the lowest average
round-trip processing delay among other protocols.
Likewise, the upstream throughput results clearly show there
is no high difference between all protocols, as every protocol
almost close to each other. While according to the results of
the tests, MQTT protocol became the highest throughput
among the protocols in terms of downstream throughput.

Experimental results showed that the performance of

different protocols are dependent on different network
conditions. Also, one of the most important points that is
been observed that the header size of each protocol plays a
big role in the efficiency of the routing protocol, as a fixed
size payload is been used for all tests.

IJISRT21MAY311 www.ijisrt.com 560