3rd International Conference on Internet of Things & Connected Technologies (ICIoTCT), 2018

A Performance Evaluation of Linux Containers on similar ARM based

Internet of Things Development Boards
Senthil Kumaran. Sa,b , Balakannan. S.P.a,c
a
Department of Information Technology, Kalasalingam Academy of Research and Education, Krishnankoil. TamilNadu, India.
b
sk@stylesen.org
c
balakannansp@gmail.com

Abstract
Internet of Things (IoT) is evolving and there will be more than 50 billion connected things in the near future.
Security is of prime importance when so many devices are connected and it is an active research topic among
others, in the IoT world. We need innovative ways to deploy, manage and update the software installed
on these devices in order to keep the devices updated with security fixes delivered on time. Containers
can help in achieving a new mechanism to deploy and apply updates to these IoT devices, which many
System on a chip (SoC) vendors and other operating system vendors handling IoT devices are considering.
There are many ARM[1] based development boards available commercially off-the-shelf offering various
hardware specifications. In this paper we will use Linux containers on ARM based development boards such
as Dragonboard 410c, Orange Pi PC2, Pine A64 and Raspberry Pi 3 model B which are commonly used in IoT
development, in order to benchmark and measure their performance at various levels. The ARM development
boards are carefully chosen in order to have similar hardware specification and running a common Operating
System namely Debian[2] with Linux kernel 4.13 and above. The LXC environment used across all the boards
is based on Debian Stretch (9.0).
Keywords: internet of things, iot, containers, linux containers, lxc, security, benchmark, SoC, ARM,
measurement, commercial off-the-shelf

1. Introduction the software running on the device updated. Taking

into consideration the number of devices that will
Availability of commercial off-the-shelf ARM be deployed in future, traditional software delivery
based development boards help many researchers mechanisms may not scale. Many vendors involved
and start-up companies to prototype and implement in developing IoT solutions are considering innova-
Internet of Things (IoT) based projects and exper- tive ways to deploy, manage and update the soft-
iments. Commercial off-the-shelf devices are con- ware installed on these devices. One such innova-
sidered as black boxes and importance is not given tive method is to use containerization within these
for performance, scalability and reliability[3]. Also, devices to deliver software.
it is important to measure the performance of these
devices in order to use it in a better way and avoid Consider a wrong software update that has been
serious implications during deployment[4]. pushed to one of these device and causing a disrup-
IoT is a growing area in computer science which tion which may be unacceptable in most scenarios.
will eventually lead to proliferation of devices. Hence there should be mechanisms to rollback the
When large number of IoT devices get connected updates that were applied and bring the device to
over the internet, security is of prime importance. the previous working condition. In order to support
The most common way to secure a device is to keep this rollback mechanism, containers can be of much

ELSEVIER-SSRN INFORMATION SYSTEMS & EBUSINESS NETWORK ISSN: 1556-5068

HTTPS://WWW.SSRN.COM/LINK/3RD-ICIOTCT-2018.HTML
1018

Electronic copy available at: https://ssrn.com/abstract=3178682

 3rd International Conference on Internet of Things & Connected Technologies (ICIoTCT), 2018

help. Instead of running software on the devices di- 2. Related Work

rectly, they can run on containers and each new up-
date will be deployed to a staging container within In the paper, “Benchmarking Internet of Things
the device before they are moved to work as the pri- devices”[3] commercial off-the-shelf IoT devices
mary container, running the service. This container such as Raspberry Pi, BeagleBoard-MX and Beagle-
based mechanism for software updates are popular Bone Black were bench-marked with the help of a
among many SoC vendors and other operating sys- cross platform benchmarking tool called lmbench[6]
tem vendors handling IoT devices. and a Constrained Application Protocol (CoAP)[7]
library based custom benchmarking tool. There
were two types of benchmarking done such as micro
benchmark with lmbench and macro benchmark us-
ing the CoAP based custom tool. The benchmarks
were based on memory performances, latency and
CoAP request round-trip times.
The IoT devices taken into consideration in this
paper[3] had differences in terms of hardware spec-
ification, specifically on the CPU where a Beagle-
Bone Black had a dual core CPU and Raspberry Pi
had a single core CPU. It wasn’t clear on the exact re-
visions or version of the boards that were taken into
consideration, which did had differences in memory
and CPU clock frequencies, specially the Raspberry
Pi’s.
Figure 1: Linux Containers (LXC) The benchmarking tool lmbench used to carry out
benchmarks is old[6] and is not under active devel-
opment. There were differences in the Linux kernel
At a high level the host operating system deployed versions such as 3.8.x and 3.6.x used in these boards
on these devices will be a minimal operating system for the benchmarks. Also the operating system envi-
that will be capable of running containers. The con- ronments had Graphical User Interface in some cases
tainers will run on top of these host operating sys- and didn’t in other cases. As discussed in the conclu-
tem in order to deliver various services that the IoT sion of this paper[3], due to the major differences in
device is intended for. The host operating system choice of devices with different hardware specifica-
should be hardened and will not be exposed to the tion, software and operating systems used for bench-
outer world and acts as a management commodity in marks, devices with better and bigger hardware spec-
order to manage containers that will get installed on ifications showed good performance than the lower
top of it. Figure 1, shows the architecture of Linux equipped devices.
containers (LXC)[5], which shares the same operat- “Evaluating Performance of Containerized IoT
ing system kernel as the host and provides isolation Services for Clustered Devices at the Network
of operating systems on the host. Edge”[8] discusses employing lightweight virtual-
ization technologies such as containerization on IoT
In this paper we will be measuring both bare metal
devices at the network edge. A framework based on
and within a container environment performance of
direct interaction between two devices and another
the chosen, similar ARM based development boards.
framework based on interaction between two devices
This will help us to identify the capabilities of these
with a supervising manager device in between were
boards and also the potential bottlenecks in using
considered for performance evaluation. It discusses
containerization, if any.
various aspects of using containerization in IoT de-
vices. Application oriented Docker[9] containers

ELSEVIER-SSRN INFORMATION SYSTEMS & EBUSINESS NETWORK ISSN: 1556-5068

HTTPS://WWW.SSRN.COM/LINK/3RD-ICIOTCT-2018.HTML
1019

Electronic copy available at: https://ssrn.com/abstract=3178682

 3rd International Conference on Internet of Things & Connected Technologies (ICIoTCT), 2018

Table 1: Development Boards

Dragonboard 410c Raspberry Pi 3 Model B Orange Pi PC2 Pine A64
SoC Qualcomm Snapdragon 410E Broadcom BCM2837 Allwinner H5 Allwinner A64
CPU ARM Cortex A53 ARM Cortex A53 ARM Cortex A53 ARM Cortex A53
Cores Quad Core Quad Core Quad Core Quad Core
Clock Speed 1.2 GHz 1.2 GHz 1.3 GHz 1.2 GHz
Instruction Set ARMv8 ARMv8 ARMv8 ARMv8
RAM 1GB LPDDR3 1GB LPDDR2 1GB LPDDR3 1GB LPDDR3
eMMC 8GB - - -
Storage MicroSD MicroSD MicroSD MicroSD
Ethernet - 10/100 10/100/1000 10/100/1000
Wireless 802.11 b/g/n 802.11 b/g/n - 802.11 b/g/n (optional)
Bluetooth 4.1 4.0 - 4.0 (optional)
GPIO 40 pins 40 pins 40 pins 40 pins
Price $75 USD $35 $20 $19

were used throughout the experiments to evaluate ware that are identical or close to identical to bring
performance. Docker is a Linux containerization a fair comparison rather than to compare apples with
technology that emerged from LXC in the past and oranges.
later moved to its own implementation methodology. In the recent years, container technology has be-
Yet, both Docker and LXC use the same underly- come popular in GNU/Linux operating systems with
ing mechanisms offered by Linux Kernel in order to Linux Containers (LXC). We will use LXC as our
implement containerization, hence Docker could be container technology.
considered an equivalent and similar technology to
LXC. Thorough analysis of different parameters such 3.1. Device Configuration
as CPU utilization, power efficiency, system load,
The following similar ARM based development
CoAP response times, etc. were made. The IoT de-
boards are chosen for this paper:
vices used in these experiments were same and were
running Docker based containers to measure other
1) Dragonboard 410c [12]
system and network parameters.
In the paper, “A Performance Evaluation of 2) Orange Pi PC2 [13]
Container Technologies on Internet of Things 3) Pine A64 [14]
Devices”[10] the performance evaluation was done 4) Raspberry Pi 3 Model B [15]
on a single device namely the Raspberry Pi 2[11]
both on bare metal and within container environ- These boards are popular among IoT developers
ment. This paper also used Docker as the con- and are also affordable with similar hardware specifi-
tainerization technology and analysed benchmark re- cation. The detailed specification of these boards are
sults for CPU, Network I/O, Memory, Disk I/O and given in Table 1. As seen in Table 1, all these boards
Apache server. It used tools such as sysbench[18], have a quad core ARM Cortex A53 CPU which has a
dd[20] and apache benchmark (ab) tool to conduct 64-bit architecture and uses ARMv8 instruction set,
the benchmarks. It concludes with a note on negli- though these CPUs come from different SoC ven-
gible impact seen on employing containerization on dors such as Qualcomm, Broadcom and Allwinner.
Raspberry Pi 2 board that was considered. The CPUs have the same clock speeds at 1.2 GHz.
Similarly the RAM size is same across the boards at
1GB. In the benchmark measurements Ethernet was
3. Proposed Model used for networking which was connected to a Gi-
gabit LAN network. For boards which did not have
Throughout this paper care has been taken to run Ethernet, QuantumZERO QZ-AD02 USB 3.0 Giga-
performance measurements on hardware and soft- bit Ethernet Adapters[16] were used.

ELSEVIER-SSRN INFORMATION SYSTEMS & EBUSINESS NETWORK ISSN: 1556-5068

HTTPS://WWW.SSRN.COM/LINK/3RD-ICIOTCT-2018.HTML

1020
 3rd International Conference on Internet of Things & Connected Technologies (ICIoTCT), 2018

One of these boards namely the Dragonboard 410c 4.2. netperf

has inbuilt-eMMC, but the rest of them did not have Netperf is a benchmark that can be used to mea-
an eMMC. Hence to make a fair analysis, we have sure various aspects of networking performance. It
installed the Operating system on a SanDisk 16GB can be used to measure both TCP and UDP perfor-
Class 10 microSD Card. mances. [19]
3.2. Operating System and Kernel 4.3. dd
Care has been taken to run the same Operating sys- dd is a command-line utility for Unix and Unix-
tem platform - Debian aarch64 or arm64 for measur- like operating systems whose primary purpose is to
ing the performance parameters. Armbian[17] which convert and copy files. [20]
is a derivative of Debian is used for some boards such
as Orange Pi PC2 and Pine A64. All these boards run
Linux kernel version greater than or equal to 4.13.x, 5. Results and Discussion
which is the latest available at the time of this writ-
ing. This section details the benchmark results on the
four different ARM development boards that are
3.3. Performance Parameters listed in Table 1. Experiments were carried on bare
This paper focuses on measuring both bare metal metal hardware to measure the performance followed
and within LXC performance of the following pa- by measurements within the container. Each bench-
rameters: mark was run thrice and the average of the three runs
were taken as the final result value. Finally, results
a) CPU Performance are presented and discussed.
b) Memory Performance
c) Network Performance 5.1. CPU Performance Analysis
d) I/O Performance The CPU performance benchmark was done us-
ing sysbench with 4 worker threads created. This
benchmark performs prime number calculation for
each request using 64-bit integers, upto the maxi-
4. Tools Analysis mum prime number specified as 20000. The lower
the time taken for execution in seconds is better. The
benchmark was run on each of the bare metal devel-
Table 2: Benchmarking Tools Version opment boards and within LXC.
Tool Version
sysbench 0.4.12-1.2
netperf 2.6.0-2.1 Table 3: CPU Performance Benchmark Completion Time in
dd 8.26 (from coreutils) seconds
Board Bare Metal Inside LXC
Dragonboard 410c 7.50 7.49
Following sections provide the list of Benchmark- Orange Pi PC2 7.86 7.85
ing tools that were used to measure different perfor- Pine A64 11.11 11.11
mance related parameters. Table 2, provides the ver- Raspberry Pi 3 Model B 15.28 15.28
sion of each of these benchmarking tools.
Figure 2, shows that the bare metal CPU perfor-
4.1. sysbench mance matches the CPU performance measurements
Sysbench is a benchmark tool used for evaluat- taken within LXC, with negligible difference in mil-
ing CPU-bound operations and memory related per- liseconds. Dragonboard 410c completed the bench-
formance. It is modular, cross-platform and multi- mark in the lowest time around 7.5 seconds and
threaded [18]. Raspberry Pi 3 Model B took the longest time around

ELSEVIER-SSRN INFORMATION SYSTEMS & EBUSINESS NETWORK ISSN: 1556-5068

HTTPS://WWW.SSRN.COM/LINK/3RD-ICIOTCT-2018.HTML

1021
 3rd International Conference on Internet of Things & Connected Technologies (ICIoTCT), 2018

Figure 2: CPU Performance Benchmark Figure 3: Memory Performance Benchmark

15.2 seconds. The detailed time taken for CPU per- and within LXC run of the memory performance
formance benchmark completion is listed in Table 3 benchmark, in the order of 50+ seconds.
for each development board.
5.3. Network Performance Analysis
5.2. Memory Performance Analysis
The Network performance benchmark was done
Memory performance analysis was done with the
using netperf tool which is capable of perform-
help of sysbench in order to measure the memory
ing various network related benchmarking both with
operations speed. All the development boards have
TCP and UDP protocols. As listed in Table 1, not all
1GB of RAM as listed in Table 1. Each mem-
the development boards discussed in this paper has
ory block size used in the performance benchmark
an onboard Ethernet interface, which is also true for
measurement is 1 KiB. Write operations were per-
the WiFi interface. Also, a closer look shows Rasp-
formed by sysbench taking the entire 1GB of avail-
berry Pi 3 Model B has an onboard Ethernet interface
able memory into account. The sysbench com-
that supports 10/100 and other boards which have on-
mand was started with 4 worker threads. In Fig-
board Ethernet interface supports 10/100/1000 band-
ure 3, we can see that Pine A64 development board
width. For Dragonboard 410c which does not have
took the longest time to execute the memory bench-
an onboard Ethernet we have provided an USB Eth-
mark,whereas Dragonboard 410c took the least time
ernet dongle to get it connected to the network. In the
for execution. The details of total time taken for ex-
case of Pine A64, we could not make the onboard
ecution is given in Table 4.
Ethernet working, hence we added a USB Ethernet
dongle and used the same for running the benchmark.
Table 4: Memory Performance Benchmark Completion Time QuantumZERO QZ-AD02 USB 3.0 Gigabit Ethernet
in seconds
Board Bare Metal Inside LXC Adapters[16] were used in both the cases. The LXC
Dragonboard 410c 35.52 35.05 containers running inside the development boards are
Orange Pi PC2 36.13 36.13 connected via a virtual bridge formed with the help
Pine A64 86.80 140.07 of libvirtd[21].
Raspberry Pi 3 Model B 63.94 64.48 Netperf requires a netserver service running on a
different machine to which the netperf client sends
Also it is worth noting that there is considerable data packets and measures the throughput of the net-
difference in execution time of Pine A64 bare metal work. The netserver service was run on a laptop

ELSEVIER-SSRN INFORMATION SYSTEMS & EBUSINESS NETWORK ISSN: 1556-5068

HTTPS://WWW.SSRN.COM/LINK/3RD-ICIOTCT-2018.HTML

1022
 3rd International Conference on Internet of Things & Connected Technologies (ICIoTCT), 2018

Figure 4: Network Throughput

Table 5: Network Throughput in MB/s

Board Bare Metal Inside LXC
TCP UDP TCP UDP
Dragonboard 410c 113.18 98.21 105.99 98.94
Orange Pi PC2 913.98 675.04 315.82 318.74
Pine A64 123.43 131.11 122.42 131.45 Figure 5: I/O Performance Benchmark
Raspberry Pi 3 Model B 94.10 94.00 66.49 90.24
Table 6: I/O Benchmark Completion Time in seconds
Board Bare Metal Inside LXC
which was a single hop away from each of the de-
Dragonboard 410c 9.75 9.85
velopment boards. The laptop and all the develop- Orange Pi PC2 43.86 43.94
ment boards were connected using a Gigabit switch Pine A64 10.60 10.88
with CAT6[22] cable, which ensures the network can Raspberry Pi 3 Model B 56.11 56.83
have a maximum throughput of 1 GB/s or 1024 MB/s
(theoretically).
which serves as the storage for all the development
The measurement was taken by sending a test boards. The I/O performance analysis was done by
packet of size 1024 bytes each to the remote host, creating a 100 MiB file by copying 1024 KiB at a
first using the TCP, followed by sending UDP pack- time for 100 times using dd command. The flags
ets of the same size. The benchmark was run on both such as direct and dsync were used in the dd com-
bare metal and within LXC setup of development mand to avoid any caching on the memory.
boards and the results are shown in Figure 4, where
The time taken in seconds to complete creating a
the Y-axis represents throughput in MB/s. Orange Pi
100 MiB file was measured in each case as shown
PC2 has an onboard Gigabit Ethernet adapter which
in Table 6. The lower the execution time, is better,
was used directly in the analysis and that did show
in this case. As seen in Figure 5, Dragonboard 410c
peak throughput of around 900+ MB/s. All the other
took less than 10 seconds to completely write a 100
development boards showed a throughput of 10/100
MiB file, whereas Raspberry Pi 3 Model B took the
analogous bandwidth. There was not much differ-
longest time i.e., 56 seconds to create the same file
ence in throughput observed between bare metal and
using dd.
within LXC executions, except in the case of Orange
Pi PC2. Table 5, shows the measured value of net-
work throughput.
6. Future Work
5.4. I/O Performance Analysis The development boards considered in this paper
As noted already, some of the development boards are mostly employed as IoT Gateways / Prototypes
has an onboard eMMC and some does not. In or- and Edge devices, since they have considerably big-
der to overcome this difference the Operating Sys- ger resource in terms of CPU, RAM, connectivity,
tem was installed on identical 16 GB MicroSD card, etc. which are typically not the case of constrained

ELSEVIER-SSRN INFORMATION SYSTEMS & EBUSINESS NETWORK ISSN: 1556-5068

HTTPS://WWW.SSRN.COM/LINK/3RD-ICIOTCT-2018.HTML

1023
 3rd International Conference on Internet of Things & Connected Technologies (ICIoTCT), 2018

IoT devices[23] classified as class0, class1, class2. Clustered Devices at the Network Edge”, IEEE INTER-
In such non-constrained devices it is important to NET OF THINGS JOURNAL, VOL. 4, NO. 4, pp. 1019-
analyse the application protocols and measure the 1030, 2017.
[9] Docker. https://www.docker.com/; 2018.
performance of applications that will be deployed in [10] Morabito Roberto, “A Performance Evaluation of Con-
real world. Future work will focus on evaluating the tainer Technologies on Internet of Things Devices”, IEEE
performance of constrained IoT devices and applica- Access, VOL. 5, pp. 8835-8850, 2017.
tions run as service in individual containers. [11] Raspberry Pi 2 http://www.raspberrypi.org/
products/raspberry-pi-2-model-b/ 2018.
[12] Dragonboard 410c https://www.qualcomm.com/
products/dragonboard-410c 2018.
7. Conclusions [13] Orange Pi PC2 http://www.orangepi.org/
orangepipc2/ 2018.
The principal aim of this paper was to evaluate [14] Pine A64 https://www.pine64.org/?page_id=1194
the performance of few of the popular commercial 2018.
off-the-shelf development boards and look for any [15] Raspberry Pi 3 Model B https://www.raspberrypi.
org/products/raspberry-pi-3-model-b/ 2018.
performance degradation in terms of employing con- [16] QuantumZERO QZ-AD02 USB 3.0 Gigabit Ethernet
tainers on the development boards. The results that Adapter. https://goo.gl/ptazT8, 2018.
were shown and discussed clearly brought out the [17] Linux for ARM development boards. https://www.
fact that, employing Linux containers does not de- armbian.com/; 2018.
[18] SysBench, a modular, cross-platform and multi-threaded
grade the performance of the development boards
benchmark tool. https://github.com/akopytov/
in any way and provides equivalent bare metal per- sysbench; 2018.
formances. Hence Linux containers can be used to [19] Netperf, is a benchmark that can be used to measure
deliver and deploy software on IoT devices in order the performance of many different types of networking.
to provide security and ensure roll back possibilities https://hewlettpackard.github.io/netperf/;
2018.
during software updates, as discussed in the start of
[20] dd, UNIX command line tool to convert and copy files.
this paper. This paper will also stand as a piece of https://en.wikipedia.org/wiki/Dd_(Unix); 2018.
evidence to support choosing the right development [21] Virtualization APIs. https://libvirt.org/; 2018.
boards based on the results discussed for researchers [22] Category 6 cable, commonly referred to as Cat 6. https:
and start-up companies who are involved in IoT. //en.wikipedia.org/wiki/Category_6_cable;
2018.
[23] C. Bormann, M. Ersue, A. Keranen RFC7228, Terminol-
REFERENCES ogy for Constrained-Node Networks Abstract 2014.

[1] ARM architecture. https://www.arm.com/; 2018.

[2] Debian - The Universal Operating System. https://www.
debian.org/; 2018.
[3] Kruger, C. P., and Hancke, G. P. (2014). Bench-
marking Internet of things devices. Proceedings
- 2014 12th IEEE International Conference on
Industrial Informatics, INDIN 2014, 611-616.
https://doi.org/10.1109/INDIN.2014.6945583
[4] J. Villasenor and M. Tehranipoor, “Chop shop electronics”,
IEEE Spec- trum, vol. 50, no. 10, pp. 41-45, 2013.
[5] Linux Containers (LXC) https://linuxcontainers.
org/ 2018.
[6] McVoy Larry, Staelin Carl, “lmbench: Portable Tools for
Performance Analysis”, Proceedings of the USENIX An-
nual Technical Conference, 1996.
[7] Z. Shelby, K. Hartke, C. Bormann IETF RFC7252, The
Constrained Application Protocol (CoAP), 2014.
[8] Morabito Roberto, Farris Ivan, Iera Antonio, Taleb Tarik,
“Evaluating Performance of Containerized IoT Services for

ELSEVIER-SSRN INFORMATION SYSTEMS & EBUSINESS NETWORK ISSN: 1556-5068

HTTPS://WWW.SSRN.COM/LINK/3RD-ICIOTCT-2018.HTML
1024