Received December 10, 2019, accepted December 19, 2019, date of publication December 23, 2019,

date of current version January 9, 2020.

Digital Object Identifier 10.1109/ACCESS.2019.2961511

Amazon, Google and Microsoft Solutions for IoT:

Architectures and a Performance Comparison
PAOLA PIERLEONI , ROBERTO CONCETTI , ALBERTO BELLI , AND LORENZO PALMA
Dipartimento di Ingegneria dell’Informazione, Università Politecnica delle Marche, 60121 Ancona, Italy
Corresponding author: Roberto Concetti (r.concetti@pm.univpm.it)

ABSTRACT Internet of Things (IoT) aims to connect the real world made up of devices, sensors and
actuators to the virtual world of Internet in order to interconnect devices with each other generating
information from the gathered data. Devices, in general, have limited computational power and limited
storage capacity. Cloud Computing (CC) has virtually unlimited capacity in terms of storage and computing
power, and is based on sharing resources. Therefore, the integration between IoT and CC seems to be one of
the most promising solutions. In fact, many of the biggest companies that offer Cloud Services are focusing
on the IoT world to offer services also in this direction to their users. In this paper we compare the three
main Cloud Platforms (Amazon Web Services, Google Cloud Platform and Microsoft Azure) regarding
to the services made available for the IoT. After describing the typical architecture of an IoT application,
we map the Cloud-IoT Platforms services with this architecture analyzing the key points for each platform.
At the same time, in order to conduct a comparative analysis of performance, we focus on a service made
available by all platforms (MQTT middleware) building the reference scenarios and the metrics to be taken
into account. Finally, we provide an overview of platform costs based on different loads. The aim is not
to declare a winner, but to provide a useful tool to developers to make an informed choice of a platform
depending on the use case.

INDEX TERMS AWS, Azure, Cloud Computing, Cloud-IoT, Google Cloud Platform, Internet of
Things, MQTT.

I. INTRODUCTION capacity in terms of storage and computing power, and is

Internet of Things (IoT) is an Internet-based paradigm that based on sharing resources and maximizing them. CC is a
includes several interconnected technologies for the informa- model that allows access to a set of shared and configurable
tion exchange between devices, generally small ‘‘things’’ of computing resources (e.g. networks, servers, storage struc-
the real world, that can be identified and monitored through tures, applications) offered as services. These resources can
the Internet. IoT applications must take into account different be quickly requested, managed and used in a pay-as-you-
factors depending on the application context. Data produced go model, so the user pays for the amount of effective use
by things must be processed, interpreted, stored and each of a resource. CC is also location independent, allowing the
implementation choice is important for the success of an user’s access to cloud services from any location and with any
application, such as choosing the best Data Base Management device through the internet connection.
Systems (DBMS) for storing the sensed data [1]. Things In literature, the focus on CC and IoT and their integration
are widely distributed and they usually have limited storage is growing:
capacity and processing, with problems concerning reliabil- • Botta etal. [4], [5] have demonstrated the effective com-
ity, performance, security and privacy. Similar problems that plementarity of IoT and CC in terms of communication,
have been found in Mobile Computing (e.g storage, band- storage and computation [5].
width, scalability) [2]. These limits lead to an integration with • The need for integration of Cloud and IoT is presented
Cloud Computing (CC) [3], which has virtually unlimited in [6]. The authors assert that the Cloud will act as
intermediate layer between the applications and the
The associate editor coordinating the review of this manuscript and things, concealing all the functionality and complexities
approving it for publication was Yuan Zhuang . required for processing.

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see http://creativecommons.org/licenses/by/4.0/
VOLUME 8, 2020 5455
 P. Pierleoni et al.: Amazon, Google and Microsoft Solutions for IoT: Architectures and a Performance Comparison

• The study in [7] shows how Cloud-Assisted Remote • Al-khafajiy et al. in [19] focus on the resource manage-
Sensing (CARS) enables distributed sensory data col- ment and monitoring and how to balance the Fog load
lection, global resource and data sharing, remote and in order to avoid delays for real-time systems and to not
real-time data access, elastic resource provisioning and get the QoS levels worse.
scaling, and pay-as-you-go pricing models, underlining • In [20] authors present the improved previous frame-
potentials for enabling the so-called Internet of Every- work with optimal management of resources as far as
thing (IoE). job allocation in an Healthcare scenario.
• The paper [8] mainly focuses on a common approach to It is possible to consider Fog Computing as a connection
integrate the IoT and CC under the name of CloudThings point between devices and the Cloud, in fact all the mentioned
architecture and examine an IoT-enabled smart home studies agree on the central role of the Cloud. Different Fog
scenario to analyze the IoT application requirements. nodes flow the data to the Cloud. Therefore, studying the
• A solution for merging IoT and Cloud is proposed by performance of a Cloud platform, it is also useful to make
Nastic et al. [9]. They argue that system designers and weighted decisions on when and how to offload the Fog
operations managers face numerous challenges to real- nodes.
ize IoT Cloud systems in practice, due to the com- As reported in [21], with the increasing use of the
plexity and diversity of their requirements in terms of Cloud, depicting the performance of Cloud services become
IoT resources consumption, customization and runtime a priority. Many different performance tests that could be
governance. performed.
• Liu et al. [10] propose a data-centric IoT framework that • In [22] authors discuss the evaluation of different
takes advantages of the Azure public Cloud to realize Infrastructure as a Service (IaaS) Cloud systems, using
a centralized IoT management service, extending their micro-benchmarks.
previous work [11], where they present a remote mon- • Shukla et al. [23] propose RIoTBench, a real-time IoT
itoring and management solution, that is specific to a benchmark suite used to evaluate Distributed Stream
plant wall system, based on Cloud and IoT. Processing System for IoT applications hosted in Cloud
• As described in 2013 by Gubbi et al. [12] IoT anc CC data centers.
can converge. They present a Cloud centric vision for • CloudHarmony [24] offers a large collection of bench-
worldwide implementation of IoT and a Cloud imple- marks and users are able to build their own set of bench-
mentation using Aneka, which is based on interaction of marks and reports. This solution can compare Cloud
private and public Clouds. Providers and services, e.g. compute engine, storage,
On the global market the first IoT platforms integrated DNS, CDN, but not IoT related services.
with CC start being developed, in parallel with studies on According to our best knowledge, there are no studies that
IoT-Cloud native platforms. compare a typical architecture of an IoT application with
• The paper [13] presents a survey that identifies several the related services that a selection of Cloud-IoT platforms
service domains IoT Cloud platforms should deal with. makes available. There are also no studies comparing the
• In [14] the authors propose a framework for evalu- performance of the selected platforms on the basis of the IoT
ating the IoT platforms from the perspective of how services made available.
widely they cover the potential needs of the application There are hundreds of IoT platforms available from a range
providers, and their results suggests how none of the of vendors. Some platforms are highly specialized, other are
platforms today offers comprehensive support. focused on providing only a subset of the functions that
• Pflanzner and Kertész in [15] introduce a taxonomy of might be required for an IoT system. According to ‘‘The
IoT application properties and investigate 23 IoT Cloud IoT Developer Survey 2018’’ [25] drawn up by the Eclipse
use cases performing a detailed classification of them in IoT Working Group, in collaboration with the Agile-IoT
a survey. H2020 Project, IEEE, and the Open Mobile Alliance (now
IoT devices produce a large amount of data that must OMA SpecWorks), the main general-purpose end-to-end
be transmitted for processing. Traditional CC architectures Cloud-IoT platforms that currently have the leadership on the
may not meet requirements in terms of latency and real-time global market are Amazon Web Service (AWS), Microsoft
decision-making approach. Therefore, according to recent Azure and the Google Cloud Platform. Our analysis will
studies [16], Fog Computing could be a solution. focus on the comparison of these Cloud platforms regarding
• Fog Computing provides computation, storage and net- the services they make available for the IoT.
working at the edge of the IoT network, between devices In order to conduct a complete comparative performance
and Cloud data centres. [17]. analysis of the three selected platforms, it is necessary to
• Bonomi et al. [18] define characteristics of Fog Com- construct a typical scenario of an IoT application. In the
puting and demonstrate the role of Fog in the IoT, high- meantime, the breadth of the application fields (e.g. smart-
lighting how Fog nodes provide localization, enabling home, smart-traffic, industrial applications, e-health) and
low latency and context awareness, and how the Cloud consequent diversity of the involved services, would make
provides global centralization. the study limited to the concerned sector. Selected Cloud

5456 VOLUME 8, 2020

P. Pierleoni et al.: Amazon, Google and Microsoft Solutions for IoT: Architectures and a Performance Comparison

platforms are all based on a common architecture for IoT

services. After a safe connection to the Cloud, devices send
their sensed data (directly or through a gateway) to the Cloud.
The same devices (as well as other applications or other
devices) receive messages resulting from any elaboration
on the Cloud. Selected platforms are all based on a pub-
lish/subscribe mechanism [26], a message-oriented middle-
ware that allows a distributed, asynchronous, and loosely
coupled communication between messages producers and
consumers.
All three selected platforms support the MQTT proto-
col. There are several studies in literature that make a qual-
itative comparison between the MQTT protocol and other
protocols applied in IoT, [12], [27]–[30], underlining how
MQTT could be one of the most suitable protocol for IoT
solutions. Other studies compare MQTT performance with
other protocols such as HTTP [31], REST [32], AMQP [33],
COAP [34] o DDS [35]. In all these studies MQTT obtains
the best results in term of end-to-end delay and bandwidth
consumption. The MQTT protocol is based on a message bro-
ker that has an intermediary role between publishing clients’
messages and receiving subscribers’ messages. The broker
uses a topic based system (hierarchical strings) in order to
send and receive messages. To receive messages on topic of
interest, clients subscribe to that topic. FIGURE 1. IoT architectures: (a) 3 layers and (b) 5 layers.
The main objectives of this study are:
• identify the key elements of a typical architecture of an architecture. Even if there is no reference model, the basic
IoT solution in the Cloud; models in literature [37] are a 3 or 5 layer architecture [38],
• map the services offered by the three platforms accord- [39]. The 3 layer architecture in Fig.1a consist of the Appli-
ing to this architecture; cation, the Network and the Perception Layer. The 5 layer
• identify comparison metrics; shown in Fig.1b add more abstraction introducing a Middle-
• conduct a comparative analysis building same scenarios ware layer between the Network and the Application, and a
and implementing them on the three platforms, in such Business layer at the top.
a way as to have results based on these metrics. The perception layer includes sensors and actuators to
The rest of this paper is organized as follows: in Section II perform different functionalities. Data generated by this layer
we present the typical architectures of an IoT solution and a is sent through the network layer, that includes technolo-
cloud-based solution, and we identify the metrics for com- gies such as RFID, 3G, GSM, UMTS, WiFi, Bluetooth Low
parison, focusing on one of the services provided, discussing Energy, infrared, ZigBee, etc, to the Middleware layer. This
the creation of scenarios for analysis and implementation layer processes received data, makes decisions, and delivers
choices; in Section III we conduct the analysis of the three the required services to the Application layer. Finally the
platforms according to a reference architecture; we focus on business layer manages the overall IoT system activities and
the performance results obtained from the simulations and on services, and supports decision-making processes based on
their analysis and finally we analyze the pricing tiers of the data.
platforms; conclusions and future developments of our work
are illustrated in Section IV. 2) CLOUD-IoT ARCHITECTURE
II. MATERIALS AND METHODS The Cloud Standards Customer Council [40] created a
The objective of the work is to conduct two parallel analysis standard reference architecture for large-scale IoT systems
on the services provided by Cloud Platforms for IoT. On the that utilize CC. The ‘‘Cloud Customer Architecture for
one hand an analysis on the key points of the platforms, on the IoT (CCAIoT)’’ [41] covers an end-to-end solution ranging
other a performance analysis of platform service times and from the users of IoT devices to the enterprises managing
related costs. the devices [42]. The core components of the architecture
depicted in Fig.2 across five domains: User Layer, Prox-
A. ARCHITECTURES imity Network, Public Network, Provider Cloud, and Enter-
1) IoT ARCHITECTURE prise Network. User Layer contains IoT users and their
The increasing number [36] of heterogeneous connected and end user applications. It is independent of any specific
interconnected objects make the need for a flexible layered network domain. The Proximity Network domain has

VOLUME 8, 2020 5457

 P. Pierleoni et al.: Amazon, Google and Microsoft Solutions for IoT: Architectures and a Performance Comparison

FIGURE 3. High level cloud-IoT 3-tier architecture.

3) CLOUD-IoT PLATFORMS
FIGURE 2. Elements of IoT solutions. (Source: Cloud standards customer
council, Cloud customer architecture for IoT, 2016). According to the previous depicted architecture, Cloud-IoT
platforms provide middleware to connect and manage hard-
ware devices and the data collected by them. IoT solutions
networking capabilities. Devices (including sensor/actuator, in the Cloud require secure, bidirectional communication
firmware and management agent) and the physical entity are between devices and a solution back end. An IoT application
part of the Proximity Network domain. The Public Network involves many heterogeneous IoT devices, with sensors that
interconnects the devices of different proximity networks produce and send data or events through a network. They can
through a wide area network, typically the Internet. It also be used to generate insights (e.g. data processing and ana-
contains Edge Service, that allows the safe flow of data from lytics). In the rest of this section we investigate the platform
the Internet into the Provider Cloud. The Provider Cloud tier and the enterprise tier of the Cloud-IoT architecture and
captures data from devices or other data source and pro- we map components and functionalities in each tier with the
vides core IoT applications and associated services (storage, selected Cloud Platforms.
analytics, visualization). It includes components for device The keys that we consider to perform the analysis are:
management (provisioning, remote administration, software • Device management. Managing IoT hardware devices
updating, remote control). The insights generated by the is one of the core functions of an IoT platform.
Provider Cloud are used by user and enterprise application in It includes provisioning of new devices, monitoring and
the Enterprise Network domain. IoT governance and security maintenance of operating devices. IoT platforms have to
subsystems span all elements of the architecture. The secu- provide features for managing device registration, con-
rity system has to consider identity and access management figuration, over-the-air updates, and asset management,
(IAM), data protection, security monitoring, analysis and including the ability to list and search connected devices
response. and to query and manage device metadata.
The Cloud components of IoT architecture are in a • Data communication protocols. In order to enable the
three-tier pattern according to Industrial Internet Consortium remote management of devices and data transmis-
Reference Architecture [43] as depicted in Fig.3. This archi- sion, a secure and reliable communication between
tecture comprises edge, platform and enterprise layers. The IoT devices, gateways, and cloud-based apps and ser-
edge tier includes Proximity and Public Networks of the vices is essential. In literature [46] these communi-
reference architecture where data is collected from devices cation types for IoT environment are usually referred
and transmitted to devices. A device can either communicate as ‘‘Device-to-Device (D2D)’’, ‘‘Device-to-Application
directly, or through an intermediate gateway, with the Cloud. (D2A)’’, ‘‘Device-to-Gateway (D2G)’’ and ‘‘Device-to-
The field-gateway has the role of protocol translation [44] and Cloud (D2C)’’. Some data can be stored and processed
may be able to perform local storage, filtering and processing locally (e.g. field gateway in the WSN), as well as
actions on received data before sending it to the Cloud [45]. some of the data collected from sensors and other IoT
The platform tier is the Provider Cloud. It receives, processes, devices need to be delivered to cloud services for further
and analyzes data flows both in flight and at rest from the edge processing. IoT platforms incorporate message broker
tier and provides API management and visualization. It also services to enable devices and gateways to send and
provides the capability to initiate control commands from the receive messages with low latency and at scale. Message
Enterprise Network to the Public Network. The enterprise tier broker services use standard communication protocols
implements domain-specific applications, decision support like MQTT [47], CoAP [48], or XMPP [49], and support
systems and provides interfaces to end-users. It receives data web sockets.
flows from the edge and platform tier and also issues control • Rules. Once data is ingested to the IoT back-end,
commands to the platform tier and edge tier. the flow of data processing may vary, depending on

5458 VOLUME 8, 2020

P. Pierleoni et al.: Amazon, Google and Microsoft Solutions for IoT: Architectures and a Performance Comparison

scenarios and applications. Regardless, IoT platforms delay, that is the time elapsed between producing and sending
may offer a rules engine with conditions used to trigger a message by a publisher client and receiving that message
actions. by a subscriber client or application. Nevertheless end-to-end
• Data storage. Data generated by IoT devices need to delay is affected by factors, such as transmission time, prop-
be stored in order to be accessed by further processing. agation time, procession and queuing time [55]. These con-
Data is usually split in hot and cold data stores [50]. ditions are not always stable during the tests run at different
Hot data stores need to be accessed with high frequency times. Therefore, what we consider is only the cloud-broker’s
and low latency, whereas cold data stores are accessed service time in different conditions, which we illustrate in the
infrequently generally with high latency and with lower construction of the scenarios in Sec.II-B2. We also analyze
storage costs still preserving historical data. the cost of the involved services, extending a recent work by
• Integration. IoT platforms provide integration with other Kalmar and Kertesz [56] where an investigation of different
platforms, devices, web services, tools, and applications IoT Cloud service providers pricing models is conducted.
through SDKs and APIs. They construct 2 scenarios and perform the costs estimation.
• Security. This includes securing devices and network In this work we conduct a comparative analysis with a vari-
communication in addition to implementing Cloud and able number of devices (and consequently a changing number
application-level security. of messages) connected to the platforms.
• Costs. In a general Cloud environment all services are The following subsection illustrates the method used to
offered in a pay-as-you-go model, in such a way that calculate the service time of the broker on Cloud platforms.
users pay only the real use of a service/resource. We con-
sider only the cost of the IoT core solution, without con- 1) CLOUD SERVICE TIME
sidering other connected services, such as storage or data Consider a simple case with only one publisher client, P1 and
analysis. one subscriber client S1, that are MQTT clients of the same
Finding the right combination of the aforementioned capa- MQTT broker in a Cloud Platform. The experiment flow is
bilities and the use case is the basis for designing an IoT shown in Fig.4: P1 publishes a message with QoS level 1 at
solution based on CC. For example in a Smart City scenario time t0 with the timestamp t0 . At t1 the message arrives at the
information in terms of privacy and data loss are not as impor- broker, that answers to P1 with an acknowledge (PUBACK);
tant as in a healthcare domain [51]. According to [5] Smart this PUBACK arrives at time t4 to P1. In the meanwhile the
City is a fragmented scenario where common challenges broker sends the message at time t2 to S1 and S1 receives the
are related to reliability, scale and timeliness, whereas in a message at t3 . The broker service time is given by (1)
healthcare IoT solution data security and privacy by users are
the main challenges [52]. Device management and analytics Tc = t2 − t1 (1)
capabilities would be much more important for an industrial
and cultural heritage [53] IoT application. They can involve All three platforms provide to users a log system that can be
thousands of sensors to monitor different parameters inside analyzed to obtain this value. Our aim is to obtain Tc without
the same environment. Next sections investigate services the use of platforms logs. It would be possible to implement
offered by the three platforms and finally we summarize them a rule in the Cloud that adds new fields to the incoming
in Table 2. message with the timestamps t1 and t2 , but this choice is not
practicable because this operation would add significant and
B. PERFORMANCE METRICS AND SCENARIOS not easily quantifiable delays, negatively affecting the final
In order to achieve messages reliability, MQTT supports result. Furthermore, it is not possible to compare client times
three QoS levels [54]. In QoS level 0 a message is delivered with platform times unless using synchronization algorithms.
at most once and no confirmation of reception is required. In addition, in order to guarantee efficiency and scalability,
In QoS level 1, every message is delivered at least once and the platforms provide a connection endpoint, but the actual
an acknowledgement of message reception is required. QoS location of the endpoint is obviously different over time,
level 2 uses a four-way handshake mechanism for the delivery making synchronization highly expensive in terms of compu-
of a message exactly once. The Cloud platforms included in tation. Therefore we implement the simulated clients in the
this study ensure support only for QoS level 0 and 1, not for same physical machine, so as to avoid any synchronization
QoS level 2. problem between hosts. Starting from algorithms of the Net-
The aim of this study is to measure the performance of work Time Protocol [57] it is possible to calculate the Round
the access point to Cloud services for IoT: message broker Trip Delay by (2)
in AWS IoT-Core, Microsoft Azure IoT Hub and the MQTT
RTD = (t3 − t0 ) − (t2 − t1 ) (2)
Bridge of the Google’s Cloud IoT-Core. As reported in a
recent survey [46], the messaging technology has the greatest The Round Trip Time on the publisher is calculated by
usage in IoT applications, and the time is one of the most using (3)
important evaluation factors. Therefore, in order to com-
pare MQTT performance we first consider the end-to-end RTTp = t4 − t0 (3)

VOLUME 8, 2020 5459

 P. Pierleoni et al.: Amazon, Google and Microsoft Solutions for IoT: Architectures and a Performance Comparison

the instant t4 in a text file. The subscriber client does the same
operation when it receives the message in the instant t3 .
We conduct 100 tests in 23 different simulations and for
each simulation we compute the Mean and the Standard
Deviation, both from Amazon’s Log and from text files
obtained from tests, that we analyze to verify the value of the
assumptions.
Starting from a base case with one client publishing one
message per second (mps) on a topic which has one sub-
scriber client, we send 1000 messages and we track the
obtained results. The sending frequency has increased from
1 mps to 1000 mps, and then the number of publisher and
subscriber clients has changed to 5 and 10 (always maintain-
ing 1:1 ratio), resulting in an increase of up to 10000 mps.
The Mean values and Standard Deviations obtained, as shown
in Table 1, are very close for each case and in all cases the
probability distributions follow identical trends. Therefore,
FIGURE 4. Experiment flow. Publisher sends a message to the broker and it is possible to state that the errors obtained by the simpli-
the broker forwards it to the subscriber. fications adopted are negligible for the final evaluation.

2) SCENARIOS
where t0 is the time when P1 publish the QoS level 1 message In order to analyze the performance in terms of processing
and t4 when the PUBACK sent by broker comes back to P1. time of the platforms’ Cloud Gateway, we consider differ-
Since the clients reside in the same machine we can hypothe- ent scenarios with different loads. Loads refer to different
size that the One Way Delays are symmetric, so the searched dimensions, such as number of publisher clients, number
value Tc is simply given by (4) of subscriber clients, number of messages exchanged, size
Tc = t2 − t1 = t3 − t4 (4) of messages, rate of messages published or consumed. We
made the following assumptions for scenarios construction:
In order to verify that these assumptions are not too sim- fixed size of the message payload equal to 150 bytes and
plified, with consequent inconsistency of subsequent results, number of topics equal to the number of publishers (i.e.
a validation of the data is carried out by simulations to be each client publishes messages on its own reserved topic).
compared with the logs obtained from a platform. We exam- In addition, each client, both publisher and subscriber, works
ine the AWS platform that provides the CloudWatch ser- by creating its own connection to the Cloud gateway. In this
vice [58] for log analysis. After creating a policy in the way connections are not shared by multiple clients, in order
IoT-Core that allows communication between the two ser- to simulate different connected devices (or applications).
vices, it is possible to analyze the generated log, a JSON file We consider 3 cases, increasing the loads for each case,
with the following format: in such a way as to be able to analyze both the service time of
{“timestamp”: “2019-08-19 10:54:07.180”, the broker for each platform, and the reliability of the limits
“logLevel”: “INFO”, imposed by the platforms themselves.
“traceId”: “xxxxxxxx-xxxx-xxxx-xxxx-xx”, For all platforms we sign free accounts, that introduce
“accountId”: “0123456789”, some limitations, especially with regard to the Microsoft
“status”: “Success”, platform. Indeed, as reported in [59], the free-tier for Azure
“eventType”: “Publish-In”, IoT Hub allows the creation of a single unit, resulting in
“protocol”: “MQTT”, a limit of 100 connections/sec accepted and a maximum
“topicName”: “/path/to/topic”, of 8000 messages/day.
“clientId”: “cliendId”, • Point to Point: in this scenario the number of publisher
“principalId”: “principalIdentification”, clients is equal to the number of subscriber clients, each
“sourceIp”: “xxx.xxx.xxx.xxx”, of which is listening on a single topic. The load is
“sourcePort”: 01234 } increased in two different ways: keeping the rate of mes-
The eventType field has the value ‘‘Publish-in’’ when a sages sent by publishers fixed to 10 mps and increasing
message arrives at the broker and the value ‘‘Publish-out’’ the number of clients (between 1 and 600), or keeping
when the broker sends it to a subscriber client. The value the number of clients fixed (100 publisher clients and
we are looking for is the difference between the timestamp 100 subscriber), varying the frequency of sending mes-
fields of the two JSON files. We write a python application sages (between 1 mps and 10 mps per client).
that simulates a publisher and a subscriber client: a client • Fan in: in this scenario, more clients publish messages
publishes a message with an Identification number and writes (each on their own topic) and a subscriber client is

5460 VOLUME 8, 2020

P. Pierleoni et al.: Amazon, Google and Microsoft Solutions for IoT: Architectures and a Performance Comparison

TABLE 1. Mean and standard deviation of cloud service time obtained from AWS log and from simulations. Times are in ms.

listening, using the wildcard #, on all topics. The loads

are changed by acting on the number of publisher clients
connected with a message sending rate equal to 10 mps
per client.
• Fan out: in this scenario, also called broadcasting, a pub-
lisher client publishes messages on a topic, where multi-
ple subscribers are subscribed. In this scenario, the cases
are simulated keeping the sending message rate fixed at
10 mps and increasing the number of clients subscribed.

III. RESULTS
A. ARCHITECTURE FIGURE 5. AWS IoT Core architecture and integration.

1) AWS IoT CORE

The architecture of the AWS IoT Core is represented in Fig. 5. It allows devices to register in bulk and to organize
Devices report their state by publishing messages on MQTT them into groups, attaching them to access policies.
topics, that have a hierarchical name in order to identify the AWS IoT provides a registry to manage things, stored
device. The message is sent to the AWS IoT MQTT Message as JSON data. Interaction with registry is possible with
Broker, which sends it to all clients subscribed to that topic. the AWS IoT console or the AWS Command Line Inter-
Each device has a shadow object (a virtual representation of face. Furthermore, the platform provides Device SDKs
the device) that is used to store and retrieve state information for Android, iOS, Java, JavaScript, C++, Python, and
in a JSON document divided into a last reported state and Embedded C that include open-source libraries.
a desired state. An application can send a request with the • Data communication protocols. Communication to and
current state of the device or with a change in its state. When from AWS IoT Core is allowed by a publish/subscribe
the message arrives to the broker, the Rules Engine provides message broker service. The message broker supports
message processing and integration with other AWS services. MQTT to publish and subscribe, and HTTPS only to
• Device management. AWS IoT Device Management is publish, both through IPv4 and IPv6. The implementa-
the service of the AWS IoT platform that allows orga- tion of the message broker is based on MQTT v.3.1.1,
nization, monitoring, and management of IoT devices. but it does not support QoS 2, and it does not allow

VOLUME 8, 2020 5461

 P. Pierleoni et al.: Amazon, Google and Microsoft Solutions for IoT: Architectures and a Performance Comparison

the connection of two or more clients with the same

client ID simultaneously. All topics that start with $ are
reserved topics, utilized for device shadow operations
(e.g. get or update state). The broker supports connec-
tions with the HTTP protocol by the use of REST API.
• Rules and analytics. The platform uses rules in order to
interact with other AWS services. Rules are composed FIGURE 6. Azure solution architecture and integration.
by a trigger written in a SQL-like syntax and one or more
actions activated. (set by an application and read by the device) and reported
• Data storage. Data ingested to the AWS IoT Core properties (set by the device and read by an application).
is required to be stored. AWS IoT Core offers the • Device management. Microsoft Azure IoT Hub Device
direct connection with Amazon DynamoDB (NoSQL Provisioning Service is a service that enables just-in-
database) and AWS S3 (Simple Storage Service), a scal- time provisioning of devices to an IoT hub, without
able storage in the AWS Cloud. requiring human intervention. Devices contact the provi-
• Integration. In order to create and interact with sioning service endpoint passing their identifying infor-
devices AWS IoT provides a Command Line Inter- mation. The service registers the device with an IoT
face (CLI), AWS IoT API to build applications using Hub and populates the desired twin device state. Fur-
HTTP or HTTPS requests and Device SDKs. AWS thermore, Azure provides device SDKs that can be used
offers services for the collection and the processing of on devices or gateways to simplify the connectivity to
data records: Amazon Kinesis Data Stream to real-time Azure IoT Hub. SDKs are available for .NET, C, Java,
process of streaming data, AWS Lambda to execute Node.js, Python and iOS.
serverless code, Amazon Simple Notification Service to • Data communication protocols. In addition to the IoT
send or receive notifications, and Amazon Simple Queue Hub, the platform offers the Event Hub Service to per-
Service to store data in a queue. mit communication with the Cloud. Both services are
• Security. Devices must have credentials to access the designed for data ingestion on a massive scale, but the
message broker and all traffic must be encrypted by IoT Hub includes more specific features for IoT context,
Transport Layer Security (TLS). Platform support as such as bi-directional communication to and from the
identity principals for authentication X.509 certificates, Cloud and device-level identity. Azure IoT Hub provides
typically used by AWS IoT devices, Identity Access support for the AMQP 1.0 with optional WebSocket
Management (IAM) users, groups and role, Feder- support,MQTT 3.1.1, and native HTTP 1.1 over TLS
ated identities used by web and desktop applications, protocols. QoS 2 delivery assurance of MQTT is sup-
and Amazon Cognito identities, in general used by ported, but not recommended due to the high impact
mobile applications, that allow the use of other identity on latency and availability of the entire system. The
providers. platform also offers the IoT Protocol Gateway to enable
• Costs. AWS bills separately for usage of Connectivity, other protocol adaptation for the IoT Hub.
Messaging, Device Shadow usage (device state storage), • Rules. Azure IoT Hub exposes its functionality by the
Registry usage (device metadata storage), and Rules use of endpoints. In order to route messages from device
Engine usage (message transformation and routing), all to these endpoints, Azure uses rules written in an SQL-
based on the region selected. like syntax evaluated on the message headers and body.
• Data storage. Regarding data storage, according to the
previous section, Azure offers services for hot and cold
2) MICROSOFT AZURE FOR IoT storage. In the hot path, data has to be available with
Microsoft Azure for IoT provides two paths: a PaaS solution lower latency than the data in a cold storage. Azure ser-
named Azure IoT solutions accelerator, and a SaaS solution vices for hot storage are Azure Cosmos DB as NoSQL
named Azure IoT Central. Both solutions utilize Azure IoT database, and Azure SQL DB as relational SQL DBMS.
Hub as Cloud gateway to securely accept data and provide Services for cold storage are Azure Blob Storage (a file
device management capabilities. The Hub is natively inte- storage database) and Azure Data Lake, a distributed
grated with other Azure cloud services and it allows secure data store. Azure offers also Time Series Insight that
bi-directional communication between devices and applica- provides analytics, storage and aggregation services.
tions. The Azure for IoT architecture is depicted in Fig. 6, • Integration. IoT Hub is natively connected with other
according to the 3 layer Cloud-IoT architecture. The message Azure services: Azure App Service, a managed plat-
sent by an authenticated device arrives to the Hub, that has form to build web and mobile apps, Notifications Hub,
built-in message routing functionalities in order to send the in order to send push notifications, and PowerBI, to cre-
message to one or more endpoint of other services. Devices ate dashboards.
have a virtual representation in the Cloud called twin device, • Security. There are three primary areas to be consid-
stored as a JSON document which contains desired properties ered regarding security: device, connection and cloud

5462 VOLUME 8, 2020

P. Pierleoni et al.: Amazon, Google and Microsoft Solutions for IoT: Architectures and a Performance Comparison

configurations, user-defined blob of data sent from the

Cloud.
• Data communication protocols. The platform supports
MQTT and HTTP for managing devices and commu-
nications. By the use of MQTT, devices send pub-
lish requests to a specific topic, whereas using HTTP,
devices do not maintain a connection to the platform,
and they send requests and receive responses. Regarding
MQTT’s Quality of Service, the MQTT Bridge supports
QoS 0 and 1.
FIGURE 7. Google cloud IoT core architecture and integration. • Rules. In order to manage data arriving at the Cloud,
the platform uses the concept of pipelines offered in
security. The Azure Hub Identity Registry provides the Google Cloud Data Flow: it allows to transform,
secure storage of device identities and each security key; aggregate, enrich and move data to other services. It is
all connections have to be initiated by the device to the also possible to operate on each published event individ-
Hub, not vice versa, and use TLS authentication with ually by the Google Cloud Functions, that can be used to
X.509 certificate; Azure Active Directory is used for filter invalid data, trigger alarms, or invoke other APIs.
user authentication and authorization for cloud access. • Data storage. Devices send different types of data, from
• Costs. Azure’s costs management is based on two levels their state (normally in a structured way), to telemetry
of service. In each level we find three tiers. Every tier data, and unstructured blobs of data (e.g. video streams).
has a daily message limit after which you will experi- In the case of structured data that identify the status of
ence throttling. The consumption of IoT Hub units is a device, the storage takes place directly in the service
measured on a daily basis, and the billing is generated provided by the IoT Core. Telemetry data arrives with
on a monthly basis. Customers are billed based on the high frequency and it has to be available in a low latency
number of IoT Hub units that have been consumed and high performance way: the platform offers Cloud
during the month. Datastore and Cloud BigQuery as NoSQL databases,
and Cloud BigTable as a fully managed data warehouse
3) GOOGLE CLOUD IoT CORE with SQL interface. The Cloud Storage is used to archive
Google solution for IoT is Google IoT Core. The main com- data used infrequently and for unstructured data.
ponents are the device manager and the protocol bridge. The • Integration. The platform provides the Google Cloud
device manager has the role of registering devices with the SDK which contains a command line tool called gcloud.
service, whereas the two protocols bridge (HTTP/MQTT) Operations are also possible by the Console and by
are used by devices in order to connect and send data to the the use of APIs client library for C#, Java, NodeJS,
Cloud. According to the Google Cloud Platform architecture GO, PHP, Python and Ruby. The Iot Core is natively
depicted in Fig. 7, devices send data to the Google IoT Core integrated with Google’s big data and machine learning
that is directly connected with Google Cloud Pub/Sub; it analysis services such as Cloud ML, Data Studio and
is an enterprise message-oriented middleware to the Cloud DataLab.
and it provides a message ingestion service. Messages are • Security. The IoT Core offers per-device public/private
then delivered to a pipeline service, the Google Cloud Data key authentication using JSON Web Tokens [60] and
Flow, that processes data and sends it to other cloud services, supports for RSA or Elliptic Curve algorithms to ver-
depending on the IoT project use case. Devices are repre- ify signatures. Concerning communications security,
sented by an ID and identified by a full resource name. It is the TLS 1.2 protocol, using root certificate authorities,
possible to define custom metadata for a device, a state which is required for MQTT connections. Google Cloud Iden-
is sent to the Cloud and a configuration, which is sent from tity and Access Management (IAM) allows to control,
the Cloud to the device. As a difference from the previous authenticate and authorize the Cloud IoT Core API
platforms, in Google solution information can be an arbitrary access.
user-defined blob of data. • Costs. The IoT Core prices are calculated on the volume
• Device management. The IoT Core Device Manager of data used in a month, over 250 MB, in three different
provides the service for managing devices. It includes levels and considering a minimum size of 1024 bytes for
registration, authentication and authorization processes. each message (for messages less than 1024 bytes, a cost
With the device manager it is possible to create and of 1024 bytes will be applied). All other services utilized
configure registries and devices within them. The device in a solution will be billed separately.
registry is configured with one or more Cloud Pub/Sub
topics to which telemetry events are published for B. PERFORMANCE ANALYSIS
all devices in that registry. A device is defined with We have simulated all clients as static on a machine with
metadata, it sends telemetry messages and receives the following features: Intel Xeon X5650 (x2) CPU, 12 MB

VOLUME 8, 2020 5463

 P. Pierleoni et al.: Amazon, Google and Microsoft Solutions for IoT: Architectures and a Performance Comparison

TABLE 2. Cloud platforms services.

cache, 2.66 GHz, 16 GB RAM with Ubuntu 18.04.1 LTS. the Paho Go Client Library [61], which allows connection to
The clients have been implemented in GoLang, a language an MQTT broker via TCP, TLS or WebSocket.
developed by Google, whose approach to concurrency differs The tool creates as many Goroutines as the clients param-
from the classic use of threads and shared memory. Concur- eter depending on the scenario considered (e.g. in scenario
rency is an intrinsic part of the Go programming language A #clients publisher and #clients subscriber, in the case B
and is managed using Goroutine and Channels. Goroutine #clients publisher and only one subscriber). Routines wait
are functions or methods that are performed concurrently for each client to be safely connected and authenticate to
with other functions or methods in the same address space, the broker, and consequently start to publish messages in
so access to shared memory must be synchronized. They are their own topic. At the same time subscribers are listening
not real threads, but light-weight threads managed entirely to that topic. Publisher clients keep track of the PUBACK
by the Go runtime. Channels are the pipes that connect timestamp received by the broker (thanks to QoS level 1) in
concurrent Goroutine. A tool has been developed that takes an array which index is the message identifier contained in
the following parameters from the command line: the MQTT the payload. In the same way, subscribers perform the same
broker endpoint (scheme://endpoint-url:port), the scenario to operation, keeping track of the reception timestamp. At the
be simulated, the number of clients, the number of messages, end of the Goroutine, the values are sent to the main routine,
the interval in ms between messages, the size of each mes- which saves the values of each publisher and subscriber client
sage, the publisher and subscriber QoS, eventual username in a bi-dimensional array. The obtained matrices are passed to
and password for the connection. In the case of connection a function that writes in a MySql database every single per-
to the Google Cloud Platform’s IoT Core, each client must formed measurement and the parameters of the simulation.
provide a JSON Web Token (JWT) for authentication. We use Cloud services performance may vary over time. Therefore,

5464 VOLUME 8, 2020

P. Pierleoni et al.: Amazon, Google and Microsoft Solutions for IoT: Architectures and a Performance Comparison

FIGURE 8. Point-to-point scenario - Cloud service times in relation to mps sent by clients. In (a) each client sends 10 mps and we change the
number of clients from 1 to 600; in (b) a fixed number of clients (100) sends messages from 1 mps to 15 mps.

we conduct 42 different measurements for each simulation

(i.e. 2/day in different times of day for 3 weeks). Subse-
quently a function computes the Mean Value of the Cloud
Service Time for each simulation and its standard deviation
and write results to the database. Finally a MatLab function
connects to the database and plots the obtained results.
1) POINT-TO-POINT
As reported in Sec. II-B2, Azure imposes limitations on the
number of accepted connections per second in the free-tier
and maximum number of messages per day accepted, as well FIGURE 9. Boxplot of AWS results.
as a maximum number of device to cloud messages equal
to 100 per second. Therefore the choice was to separate the
results obtained from this platform from the others. Simula-
tions will be repeated in the future using a pay tier.
In this scenario, the Amazon and Google platforms have a
similar and uniform behavior in terms of broker service time
in relation to the number of messages published per second.
In Fig. 8a the number of messages per second on the abscissa
is obtained by increasing the sending messages rate with
only one client connected up to the value of 100 mps, and
subsequently keeping the sending frequency of each client
fixed at 10 mps and increasing the number of connected FIGURE 10. Boxplot of GCP results.
clients between 1 and 600. Google IoT-Core responds faster
than the other platforms for almost all simulated mps values, It is interesting to note how the results of Azure (Fig. 11)
except for values between 150 and 750 mps, range during follow more symmetrical distributions than the other plat-
which Amazon provides better performance. Azure, even for forms, and how the values are more concentrated around
load conditions that can be compared to the others, has an the median. The distributions of the simulations obtained on
average service time much higher than its competitors, equal AWS and GCP are positive asymmetries, therefore with a
to 180ms ± 20ms. Platforms do not seem to be particularly greater dispersion for higher values. Finally, note the total
affected by increases in load. The limits declared by the absence of outliers for the distributions of AWS and GCP,
platforms are respected, in particular the number of accepted minimally present in Azure distributions.
connections per second per account and the number of mes-
sages per second accepted by the broker for connection. 2) FAN-IN
In Fig. 8b the load to the broker has been increased only In this type of scenario, where each client publishes on its
acting on the sending rate and fixing the number of clients own topic while a single subscriber is subscribed by wildcard
to 100. In this case the platform brokers behavior is even to all the topics, it is interesting to evaluate the message loss,
more uniform compared to the previous case, showing no i.e. the number of messages that have not been delivered to
significant changes in terms of service time. In this case, subscriber. Note that the Google and Microsoft platforms do
however, it is Amazon’s platform to achieve slightly better not allow direct subscription via wildcard, but they allow
average performance, remaining at 27ms ± 0.3ms. forwarding of all messages to an additional service: Google
In order to make a description of the distributions of per- allows registration of all virtual devices in a registry, which
formed simulations, box plots are shown in Fig.9 - 10 - 11. has a topic associated in the PubSub service [62]. Each device

VOLUME 8, 2020 5465

 P. Pierleoni et al.: Amazon, Google and Microsoft Solutions for IoT: Architectures and a Performance Comparison

FIGURE 11. Boxplot of azure results. FIGURE 13. Fan-in scenario - Cloud service time vs mps sent by clients
with 10 mps/client and from 1 to 600 connected clients.

following message bus. In this case too, these results can be

useful to the specialists of Cloud-IoT solutions for the most
correct choice based on the application needs. Simulations
in Fig. 13 shows that the service time trend is similar to that
found in the point-to-point scenario, up to 3000 mps. After
this value, Google and Amazon platforms have an important
growth in recorded time. However, analyzing the logs made
available by the platforms, the time to forward messages
from the broker to the following service does not have such
FIGURE 12. Message loss vs mps obtained with an increasing number of significant variations; these delays are attributable to the
clients and fixed 10 mps/client in fan-in scenario.
subscriber client who fails to send the PUBACK messages
to the broker, thus reaching the so-called value of ‘‘Maximum
inbound/outbound unacknowledged QoS 1 publish requests’’
sends its telemetry messages to its MQTT topic and the (due to the limitations imposed by the platforms themselves),
platform forwards them to PubSub. A client application then thus delaying the forwarding. As regards Azure, the results
simulates the connection to this service and to the topic of the are the same for the point-to-point scenario shown in Fig. 11,
registry, thus allowing to perfectly simulate the considered showing that they are in no way influenced by the different
scenario. Azure allows native integration of the IoT Hub type of scenario.
with one or more default endpoints, and also in this case
a client is connected to that endpoint to receive forwarded
3) FAN-OUT
messages. Therefore results depicted in Fig. 12 concern the
The workload in this scenario is generated by a single pro-
message loss only for Amazon by connecting directly to the
ducer publishing 10 messages per second in a single topic.
MQTT broker and subscribing to all topics. The broker is
An increasing number of clients is subscribed to that topic.
able to forward all messages to the subscriber up to the limit
We analyze the Cloud Service Time as depicted in Sec. II-
of 400 mps (40 connected clients each sending 10 mps);
B1 in function of the fan-out factor (i.e. number of subscriber
significant losses begin to occur with 70 connected clients
clients). In this scenario we have the result shown in Fig.14
(5%) with an exponential trend up to about 810 mps, and
where we can see how GCP has lower Cloud Service Time
then stabilize on 42% for a load of 1000 mps. These values
than both AWS and Azure for a fan-out factor over 15.
can be useful for developers and solution architects to make
Before this value AWS has the lowest values. Azure’s Hub
a possible partition of the topic and connected subscriber
IoT forwards messages 15X slower than the other platforms,
clients, in order to avoid significant losses of published mes-
yet having (as in previous scenarios) a lower gap between
sages. In order to make the same analysis on this scenario,
outliers. Indeed, for a fan-out factor equal to one the Cloud
we use a further service on AWS that allows forwarding
Service Time is 26.479 ms, 24.991 ms and 160.567 ms, and
all messages from a topic to AWS SNS [63], by creating
for 300 subscribers the difference is 26.7%, 68.1% and 7.1%
a rule in the Rules Engine. Also with regard to Azure the
respectively for GCP, AWS and Azure.
reasoning was the same: creation of a rule that allows the
forwarding of messages addressed to all topics of the devices
towards the message bus [64]. The subscriber client, for all C. COSTS
platforms, listens to the channel on which these messages are In this Section, according to each platform’s documentation,
forwarded and counts the actual delivery. In this case, for all we analyze costs in function of different work loads. Each
platforms, all messages were delivered in each load condi- platform has a different approach in terms of billing:
tion considered. Service Cloud Times were then calculated 1) In Amazon Web Services IoT Core connectivity,
using the procedure illustrated in Sec. II-B1, obtaining the messaging and shadow devices (device status stor-
service time of the broker added to the service time of the age), registry (device metadata storage), and rules

5466 VOLUME 8, 2020

P. Pierleoni et al.: Amazon, Google and Microsoft Solutions for IoT: Architectures and a Performance Comparison

FIGURE 14. Fan-out scenario - Cloud service times vs fan-out factor (from FIGURE 15. Costs in function of increasing number of devices.
1 to 300 subscribers client).

TABLE 3. Azure IoT Hub prizes (in $).

• Messaging is metered by the number of messages
transmitted between devices and AWS IoT Core:
$1.20 when the monthly message volume is up to
1 billion messages, $0.96 for the next 4 billion
messages, $0.84 over 5 billion messages.
2) Azure IoT Hub offers two tiers, basic and standard,
that differ in the number of features they support.
The standard tier of IoT Hub enables all features for
any IoT solutions that want to make use of the bi-
directional communication capabilities, whereas the
basic tier enables a subset of the features and is intended
TABLE 4. Google IoT core prizes (in $). for IoT solutions that only need uni-directional com-
munication from devices to the Cloud. Prizes for each
level in the two tiers are summarized in Table 3.
3) The Google IoT Core prices are calculated on the vol-
ume of data used in a month and there is no charge
for creating, reading, updating and deleting devices.
Messages below 1024 bytes are counted as 1024 bytes,
whereas for larger messages the actual size is billed.
engine (message transformation and routing) are billed Prizes are reported in Table 4.
separately. Different costs are applied according to In order to conduct an analysis of changes in cost, we con-
region/zone. Messages are measured in increments sider a scenario where devices send one message per minute
of 5 kB. For EU London zone: of 1kB. Devices are continuously connected to the plat-
• Connectivity is metered in 1 minute increments forms. Fig. 15 shows the number of devices connected on
and is based on the total time devices are connected the abscissa and on the ordinate the monthly cost in dollars
to AWS IoT Core: $0.096 (per million minutes of (the volume of monthly traffic exchanged with each platform
connection); is easily obtained by multiplying the number of devices per

TABLE 5. Platform prizes (in $) according to number of connected devices.

VOLUME 8, 2020 5467

 P. Pierleoni et al.: Amazon, Google and Microsoft Solutions for IoT: Architectures and a Performance Comparison

1440 messages/day per 30 days per kB). We use a logarithmic REFERENCES

scale for a better visualization. [1] L. Incipini, A. Belli, L. Palma, R. Concetti, and P. Pierleoni, ‘‘Databases
We analyze costs evolution varying the number of con- performance evaluation for IoT systems: The Scrovegni chapel use case,’’
in Proc. 42nd Int. Conv. Inf. Commun. Technol., Electron. Microelectron.
nected devices between 1 and 500000 (from 1440 to 720 mil- (MIPRO), May 2019, pp. 463–468.
lions of messages per day, from 1.40 MB to 687 GB per day [2] H. T. Dinh, C. Lee, D. Niyato, and P. Wang, ‘‘A survey of mobile cloud
in terms of data size). Table 5 shows the costs of the platforms computing: Architecture, applications, and approaches,’’ Wireless Com-
mun. Mobile Comput., vol. 13, no. 18, pp. 1587–1611, Dec. 2013.
in reference to some ranges of devices obtained from the [3] R. Buyya, C. S. Yeo, S. Venugopal, J. Broberg, and I. Brandic, ‘‘Cloud
analysis of the graph. We highlight for each range the lowest computing and emerging IT platforms: Vision, hype, and reality for deliv-
cost. ering computing as the 5th utility,’’ Future Generat. Comput. Syst., vol. 25,
no. 6, pp. 599–616, 2009.
[4] A. Botta, W. de Donato, V. Persico, and A. Pescapé, ‘‘On the integration
IV. CONCLUSION of cloud computing and Internet of Things,’’ in Proc. Int. Conf. Future
The growing attention of the main Cloud platforms to the Internet Things Cloud (FiCloud), Aug. 2014, pp. 23–30.
[5] A. Botta, W. de Donato, V. Persico, and A. Pescapé ‘‘Integration of cloud
IoT world is nowadays evident. With this study we wanted
computing and Internet of Things: A survey,’’ Future Gener. Comput. Syst.,
to highlight the characteristics of each platform, through a vol. 56, pp. 684–700, Mar. 2016.
mapping of services with a reference architecture, so as to [6] S. M. Babu, A. J. Lakshmi, and B. T. Rao, ‘‘A study on cloud based Internet
have a broader view of all these services that are made avail- of Things: Cloudio T,’’ in Proc. Global Conf. Commun. Technol. (GCCT),
Apr. 2015, pp. 60–65.
able. Each platform provides a Cloud access point through [7] S. Abdelwahab, B. Hamdaoui, M. Guizani, and A. Rayes, ‘‘Enabling smart
which physical devices can connect in a secure and private cloud services through remote sensing: An Internet of everything enabler,’’
way. After an authorized connection, devices can start to send IEEE Internet Things J., vol. 1, no. 3, pp. 276–288, Jun. 2014.
[8] J. Zhou, T. Leppanen, E. Harjula, M. Ylianttila, T. Ojala, C. Yu, H. Jin,
their data to the Cloud: the most used protocol is MQTT. The and L. T. Yang, ‘‘CloudThings: A common architecture for integrating
tests we conducted did not try to reach the maximum message the Internet of Things with cloud computing,’’ in Proc. IEEE 17th Int.
throughput. In fact, regarding the performance we measured Conf. Comput. Supported Cooperat. Work Design (CSCWD), Jun. 2013,
pp. 651–657.
the service time of the message broker, in different load [9] S. Nastic, S. Sehic, D.-H. Le, H.-L. Truong, and S. Dustdar, ‘‘Provisioning
conditions and in three different scenarios. At the same time, software-defined IoT cloud systems,’’ in Proc. Int. Conf. Future Internet
by increasing the load in each scenario and keeping in mind Things Cloud (FiCloud), Aug. 2014, pp. 288–295.
[10] Y. Liu, K. A. Hassan, M. Karlsson, Z. Pang, and S. Gong, ‘‘A data-centric
the use of free tiers for each platform, we were also able to Internet of Things framework based on azure cloud,’’ IEEE Access, vol. 7,
verify the limitations imposed by the providers. Results show pp. 53839–53858, 2019.
how the platforms, although with different service times, [11] Y. Liu, K. A. Hassan, M. Karlsson, O. Weister, and S. Gong, ‘‘Active plant
wall for green indoor climate based on cloud and Internet of Things,’’ IEEE
answer in a uniform way, thus guaranteeing predictable per- Access, vol. 6, pp. 33631–33644, 2018.
formance levels that comply with the specifications indicated [12] J. Gubbi, R. Buyya, S. Marusic, and M. Palaniswami, ‘‘Internet of
in the documentation. The final analysis made on the costs Things (IoT): A vision, architectural elements, and future directions,’’
Future Generat. Comput. Syst., vol. 29, no. 7, pp. 1645–1660, 2013.
of the platforms according to different types of load, allows
[13] P. P. Ray, ‘‘A survey of IoT cloud platforms,’’ Future Comput. Inform. J.,
the reader to have a further element of comparison to make vol. 1, nos. 1–2, pp. 35–46, 2016.
the choice. In fact, the work does not want one platform to [14] O. Mazhelis and P. Tyrvainen, ‘‘A framework for evaluating Internet-of-
be preferred over another, but tries to provide the developer Things platforms: Application provider viewpoint,’’ in Proc. IEEE World
Forum Internet of Things (WF-IoT), Mar. 2014, pp. 147–152.
with elements to make an informed choice. In the future the [15] T. Pflanzner and A. Kertész, ‘‘A taxonomy and survey of IoT cloud
same study will be carried out on the different paid levels that applications,’’ EAI Endorsed Trans. Internet Things, vol. 3, no. 12, p. 14,
the platforms make available, in order to have no limitations 2018.
[16] R. K. Barik, H. Dubey, C. Misra, D. Borthakur, N. Constant, S. A. Sasane,
on the number of devices or number of messages. In this R. K. Lenka, B. S. P. Mishra, H. Das, and K. Mankodiya, ‘‘Fog assisted
study we used a fixed message size and we tested the MQTT cloud computing in era of big data and Internet-of-Things: Systems,
protocol. We will study the behavior of platforms with differ- architectures, and applications,’’ in Cloud Computing for Optimization:
Foundations, Applications, and Challenges. Cham, Switzerland: Springer,
ent packets size and under different communication protocols 2018, pp. 367–394.
(e.g. HTTP or AMQP). The focal point of each platform [17] A. V. Dastjerdi and R. Buyya, ‘‘Fog computing: Helping the Internet
is the full integration with other services available (storage, of Things realize its potential,’’ Computer, vol. 49, no. 8, pp. 112–116,
Aug. 2016.
AI, Machine Learning, etc); the performance analysis carried [18] F. Bonomi, R. Milito, J. Zhu, and S. Addepalli, ‘‘Fog computing and its
out had the purpose of measuring the time of the message role in the Internet of Things,’’ in Proc. 1st Ed. MCC Workshop Mobile
broker. It would be interesting to carry out the same analysis Cloud Comput., 2012, pp. 13–16.
[19] M. Al-Khafajiy, T. Baker, A. Waraich, D. Al-Jumeily, and A. Hussain,
on all the services involved in a complete IoT application,
‘‘IoT-fog optimal workload via fog offloading,’’ in Proc. IEEE/ACM Int.
creating a common scenario to be tested on the three plat- Conf. Utility Cloud Comput. Companion (UCC Companion), Dec. 2018,
forms. In fact, in applications where response time is a critical pp. 359–364.
element, analyzing the time of each service can be interesting [20] M. Al-Khafajiy, L. Webster, T. Baker, and A. Waraich, ‘‘Towards fog
driven IoT healthcare: Challenges and framework of fog computing in
to understand where and how recover even milliseconds that healthcare,’’ in Proc. 2nd Int. Conf. Future Netw. Distrib. Syst., 2018,
can be fundamental. At the same time exploring the solutions Art. no. 9.
that the platforms make available for Fog Computing (e.g. [21] S. P. Ahuja, ‘‘On the use of system-level benchmarks for comparing public
cloud environments,’’ in Handbook of Research on Cloud Computing
AWS Greengrass) could be an interesting extension of this and Big Data Applications in IoT. Hershey, PA, USA: IGI Global, 2019,
work. pp. 24–38.

5468 VOLUME 8, 2020

P. Pierleoni et al.: Amazon, Google and Microsoft Solutions for IoT: Architectures and a Performance Comparison

[22] L. Gillam, B. Li, J. O’Loughlin, and A. P. S. Tomar, ‘‘Fair benchmarking [46] A. Souri, A. Hussien, M. Hoseyninezhad, and M. Norouzi, ‘‘A systematic
for cloud computing systems,’’ J. Cloud Comput., Adv., Syst. Appl., vol. 2, review of IoT communication strategies for an efficient smart environ-
no. 1, p. 6, 2013. ment,’’ Trans. Emerg. Telecommun. Technol., p. e3736, 2019. [Online].
[23] A. Shukla, S. Chaturvedi, and Y. Simmhan, ‘‘RiotBench: An IoT bench- Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/ett.3736, doi:
mark for distributed stream processing systems,’’ Concurrency Comput., 10.1002/ett.3736.
Pract. Exper., vol. 29, no. 21, p. e4257, 2017. [47] Information Technology—Message Queuing Telemetry Transport (MQTT)
[24] CloudHarmony, Transparency for the Cloud. Accessed: Nov. 29, 2019. v3.1.1, ISO Standard ISO/IEC 20922:2016, Jun. 2016.
[Online]. Available: http://www.cloudharmony.com/ [48] Z. Shelby, K. Hartke, and C. Bormann, The Constrained Application
[25] IoT Developer Survey 2018. Accessed: Jun. 28, 2018. [Online]. Available: Protocol (COAP), document RFC 7252, Jun. 2014. [Online]. Available:
https://www.eclipse.org/org/press-release/iotdevsurvey2018.php http://www.rfc-editor.org/rfc/rfc7252.txt
[26] D. Happ, N. Karowski, T. Menzel, V. Handziski, and A. Wolisz, ‘‘Meeting [49] P. Saint-Andre, Extensible Messaging and Presence Protocol
IoT platform requirements with open pub/sub solutions,’’ Ann. Telecom- (XMPP): Core, document RFC 6120, Mar. 2011. [Online]. Available:
mun., vol. 72, no. 1, pp. 41–52, Feb. 2017. http://www.rfc-editor.org/rfc/rfc6120.txt
[27] I. Hedji, I. Špeh, and A. Šarabok, ‘‘IoT network protocols comparison [50] J. J. Levandoski, P.-Å. Larson, and R. Stoica, ‘‘Identifying hot and cold
for the purpose of IoT constrained networks,’’ in Proc. 40th Int. Conv. data in main-memory databases,’’ in Proc. IEEE 29th Int. Conf. Data
Inf. Commun. Technol., Electron. Microelectron. (MIPRO), May 2017, Eng. (ICDE), Apr. 2013, pp. 26–37.
pp. 501–505. [51] P. Pierleoni, L. Pernini, L. Palma, A. Belli, S. Valenti, L. Maurizi,
[28] R. Sutaria and R. Govindachari, ‘‘Making sense of interoperability: Pro- L. Sabbatini, and A. Marroni, ‘‘An innovative WebRTC solution for e-
tocols and standardization initiatives in IoT,’’ in Proc. 2nd Int. Workshop health services,’’ in Proc. IEEE 18th Int. Conf. e-Health Netw., Appl.
Comput. Netw. Internet Things, 2013. Services (Healthcom), Sep. 2016, pp. 1–6.
[29] Á. Asensio, Á. Marco, R. Blasco, and R. Casas, ‘‘Protocol and architecture [52] A. M.-H. Kuo, ‘‘Opportunities and challenges of cloud computing to
to bring things into Internet of Things,’’ Int. J. Distrib. Sensor Netw., improve health care services,’’ J. Med. Internet Res., vol. 13, no. 3, p. e67,
vol. 10, no. 4, 2014, Art. no. 158252. 2011.
[30] L. Hou, S. Zhao, X. Xiong, K. Zheng, P. Chatzimisios, M. S. Hos- [53] P. Pierleoni, A. Belli, L. Palma, S. Valenti, S. Raggiunto, L. Incipini, and
sain, and W. Xiang, ‘‘Internet of Things cloud: Architecture and P. Ceregioli, ‘‘The Scrovegni chapel moves into the future: An innovative
implementation,’’ IEEE Commun. Mag., vol. 54, no. 12, pp. 32–39, Internet of Things solution brings new light to Giotto’s masterpiece,’’ IEEE
Dec. 2016. Sensors J., vol. 18, no. 18, pp. 7681–7696, Sep. 2018.
[31] T. Yokotani and Y. Sasaki, ‘‘Comparison with HTTP and MQTT on [54] S. Behnel, L. Fiege, and G. Muhl, ‘‘On quality-of-service and publish-
required network resources for IoT,’’ in Proc. Int. Conf. Control, Electron., subscribe,’’ in Proc. 26th IEEE Int. Conf. Distrib. Comput. Syst. Workshops
Renew. Energy Commun. (ICCEREC), Sep. 2016, pp. 1–6. (ICDCS Workshops), Jul. 2006, p. 20.
[32] A. A. Simiscuka, T. M. Markande, and G.-M. Muntean, ‘‘Real-virtual [55] C. Bovy, H. Mertodimedjo, G. Hooghiemstra, H. Uijterwaal, and
world device synchronization in a cloud-enabled social virtual reality IoT P. V. Mieghem, ‘‘Analysis of end-to-end delay measurements in Internet,’’
network,’’ IEEE Access, vol. 7, pp. 106588–106599, 2019. in Proc. Passive Active Meas. Workshop-PAM, 2002.
[33] J. E. Luzuriaga, M. Perez, P. Boronat, J. C. Cano, C. Calafate, and [56] E. E. Kalmar and A. Kertesz, ‘‘What does I (o) T cost?’’ in Proc. 8th
P. Manzoni, ‘‘A comparative evaluation of AMQP and MQTT protocols ACM/SPEC Int. Conf. Perform. Eng. Companion, 2017, pp. 19–24.
over unstable and mobile networks,’’ in Proc. 12th Annu. Consum. Com- [57] D. Mills, J. Martin, J. Burbank, and W. Kasch, Network Time Protocol
mun. Netw. Conf. (CCNC), Jan. 2015, pp. 931–936. Version 4: Protocol and Algorithms Specification, document RFC 5905,
[34] D. Thangavel, X. Ma, A. Valera, H.-X. Tan, and C. K.-Y. Tan, Jun. 2010. [Online]. Available: http://www.rfc-editor.org/rfc/rfc5905.txt
‘‘Performance evaluation of MQTT and COAP via a common middle- [58] Amazon CloudWatch. Accessed: Jun. 20, 2018. [Online]. Available:
ware,’’ in Proc. IEEE 9th Int. Conf. Intell. Sensors, Sensor Netw. Inf. http://aws.amazon.com/cloudwatch
Process. (ISSNIP), Apr. 2014, pp. 1–6. [59] Choose the Right IoT Hub Tier for Your Solution. Accessed: Jun. 20, 2018.
[35] Y. Chen and T. Kunz, ‘‘Performance evaluation of IoT protocols under [Online]. Available: https://docs.microsoft.com/en-us/azure/iot-hub/iot-
a constrained wireless access network,’’ in Proc. Int. Conf. Sel. Topics hub-scaling
Mobile Wireless Netw. (MoWNeT), Apr. 2016, pp. 1–7. [60] M. Jones, J. Bradley, and N. Sakimura, JSON Web Token (JWT),
[36] D. Lund, C. MacGillivray, V. Turner, and M. Morales, ‘‘Worldwide and document RFC 7519, May 2015. [Online]. Available: http://www.rfc-
regional Internet of Things (IoT) 2014–2020 forecast: A virtuous circle editor.org/rfc/rfc7519.txt
of proven value and demand,’’ Int. Data Corp., Framingham, MA, USA, [61] Eclipse Paho Go Client. Accessed: May 10, 2018. [Online]. Available:
Tech. Rep. 1, 2014. https://www.eclipse.org/paho/clients/golang/
[37] M. Wu, T.-J. Lu, F.-Y. Ling, J. Sun, and H.-Y. Du, ‘‘Research on the [62] Google Cloud Pub/Sub. Accessed: May 15, 2018. [Online]. Available:
architecture of Internet of Things,’’ in Proc. 3rd Int. Conf. Adv. Comput. https://cloud.google.com/pubsub/
Theory Eng. (ICACTE), vol. 5, Aug. 2010, pp. V5-484–V5-487. [63] Amazon Simple Notification Service. Accessed: May 15, 2018. [Online].
[38] R. Khan, S. U. Khan, R. Zaheer, and S. Khan, ‘‘Future Internet: The Available: https://aws.amazon.com/sns
Internet of Things architecture, possible applications and key challenges,’’ [64] Microsoft Azure Message Bus. Accessed: May 15, 2018. [Online]. Avail-
in Proc. 10th Int. Conf. Frontiers Inf. Technol. (FIT), Dec. 2012, pp. 257– able: https://azure.microsoft.com/services/service-bus/
260.
[39] P. Sethi and S. R. Sarangi, ‘‘Internet of Things: Architectures, proto-
cols, and applications,’’ J. Elect. Comput. Eng., vol. 2017, Jan. 2017,
Art. no. 9324035.
[40] Cloud Standards Customer Council. Accessed: Jun. 18, 2018. [Online].
Available: http://www.cloud-council.org
[41] Cloud Customer Architecture for IoT. Accessed: Jun. 18, 2018. [Online].
Available: http://www.cloud-council.org/deliverables/CSCC-Cloud-
Customer-Architecture-for-IoT.pdf PAOLA PIERLEONI received the master’s degree
[42] A. Klenik and A. Pataricza, ‘‘Performance analysis of critical services,’’ in
in electronic engineering, in 1991, and the Ph.D.
Proc. IEEE Int. Conf. Future IoT Technol. (Future IoT), Jan. 2018, pp. 1–6.
degree in electrical engineering from the Polytech-
[43] Industrial Internet Consortium Reference Architecture. Accessed:
Jun. 18, 2018. [Online]. Available: https://www.iiconsortium.
nic University of Marche, in 1995. Since 1991,
org/IIC_PUB_G1_V1.80_2017-01-31.pdf she has been with the Department of Information
[44] P. Pierleoni, A. Belli, L. Palma, L. Incipini, S. Raggiunto, M. Mercuri, Engineering, Polytechnic University of Marche,
R. Concetti, and L. Sabbatini, ‘‘A cross-protocol proxy for sensor networks where she is currently an Assistant Professor
based on CoAP,’’ in Proc. IEEE 23rd Int. Symp. Consum. Technol. (ISCT), in telecommunications. Her main research topics
Jun. 2019, pp. 251–255. include network protocols, wireless sensor net-
[45] J. Misic, M. Z. Ali, and V. B. Misic, ‘‘Protocol architectures for IoT works, the Internet of Things, signal processing,
domains,’’ IEEE Netw., vol. 32, no. 4, pp. 81–87, Jul./Aug. 2018. and embedded devices development.

VOLUME 8, 2020 5469

 P. Pierleoni et al.: Amazon, Google and Microsoft Solutions for IoT: Architectures and a Performance Comparison

ROBERTO CONCETTI received the master’s LORENZO PALMA received the master’s degree
degree in computer engineering from the Poly- (cum laude) in electronic engineering, in 2012,
technic University of Marche, Italy, in 2016. He and the Ph.D. degree in information engineering
is currently pursuing the Ph.D. degree with the from the Polytechnic University of Marche, Italy,
Department of Information Engineering, Poly- in 2017. He is currently a Research Fellow with the
technic University of Marche. His current research Polytechnic University of Marche. His research
interests include computer networks and cloud interests are the IoT, wireless sensors networks,
computing for the Internet of Things. IMU sensors, embedded systems, and communi-
cation networks. In 2017, he had a research grant
from Consortium GARR to develop a new system
for seismic and structural monitoring based on IPv6 sensors able to provide
ALBERTO BELLI received the Ph.D. degree in
data for the creation of an earthquake early warning systems.
biomedical, electronic, and telecommunications
engineering, in 2016, and the master’s degree in
telecommunications engineering from the Univer-
sità Politecnica delle Marche, Italy, in 2012. He is
currently a Staff Scientist with the Department of
Engineering, Università Politecnica delle Marche.
His main research interests are in wireless body
sensor networks for the internet of things solu-
tion, data fusion algorithms for array sensors, and
applications for biomedical devices.

5470 VOLUME 8, 2020