energies

Article
Escope: An Energy Efficiency Simulator for Internet
Data Centers
Jun Liu 1 , Longchuan Yan 1 , Chengxu Yan 2 , Yeliang Qiu 2 , Congfeng Jiang 2, * , Yang Li 1 , Yan Li 1
and Christophe Cérin 3

1 State Grid Co., Ltd., Information Communication Branch, Beijing 100761, China
2 School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou 310018, China
3 IUT de Villetaneuse, Département d’Informatique, University of Paris 13, Sorbonne Paris Cité, LIPN/CNRS
UMR 7030, F-93430 Villetaneuse, France
* Correspondence: cjiang@hdu.edu.cn; Tel.: +86-159-6810-0618

Abstract: Contemporary megawatt-scale data centers have emerged to meet the increasing demand
for online cloud services and big data analytics. However, in such large-scale data centers, servers of
different generations are installed gradually year by year, making the data center heterogeneous in
computing capability and energy efficiency. Furthermore, due to different processor architectures,
complex and diverse load dynamic changing, business coupling, and other reasons, operators pay
great attention to processor hardware power consumption and server aggregation energy efficiency.
Therefore, the simulation and analysis of the energy efficiency characteristics of data center servers
under different processor architectures can help operators understand the energy efficiency charac-
teristics of data centers and make the optimal task scheduling strategy. This is very beneficial for
improving the energy efficiency of the production system and the entire data center. The Escope
simulator designed in this study can simulate the online quantity (placement strategy) of different
types of servers in the data center and the optimal operating range of the servers. The purpose of this
is to analyze the energy efficiency characteristics of all servers in the data center and provide data
center operators with the energy efficiency and energy proportionality characteristics of different
servers, improve server utilization, and perform reasonable scheduling. Through the simulation
Citation: Liu, J.; Yan, L.; Yan, C.; Qiu,
experiment of Escope, it can be proved that running the server at the highest energy efficiency point
Y.; Jiang, C.; Li, Y.; Li, Y.; Cérin, C. or running the server under full load cannot improve the energy efficiency of the entire data center.
Escope: An Energy Efficiency The simulation algorithm provided by Escope can select the optimal set of servers and their corre-
Simulator for Internet Data Centers. sponding utilization. Escope can set up a variety of simulation strategies, and data center operators
Energies 2023, 16, 3187. https:// can simulate data center energy efficiency according to their own needs. Escope can also calculate
doi.org/10.3390/en16073187 the power cost savings of introducing new servers in the data center, which provides an essential
Academic Editors: Dimitrios reference for operators to purchase servers and design data centers.
I. Doukas and Antonios
Marinopoulos Keywords: data center; power consumption; energy proportionality; energy efficiency

Received: 8 February 2023

Revised: 23 March 2023
Accepted: 29 March 2023
1. Introduction
Published: 31 March 2023
1.1. Energy Efficiency Indicators for Data Centers
With increasing demand for cloud computing from large-scale internet applications,
enterprise-level essential services, and the construction of the digital economy, the scale
Copyright: © 2023 by the authors. and number of data centers have achieved unprecedented development. At the same time,
Licensee MDPI, Basel, Switzerland. the rapid growth in size and quantity has brought about many problems for data center
This article is an open access article
operators, such as high energy consumption, huge cost, and severe pollution [1,2]. In
distributed under the terms and
sharp contrast to the sizable energy consumption of data centers, the resource utilization
conditions of the Creative Commons
of most data centers is much lower [3–5], and the utilization of the majority of servers is
Attribution (CC BY) license (https://
generally less than 30%. Increasing server utilization can improve a data center’s energy
creativecommons.org/licenses/by/
efficiency. Placing online services and batch jobs on the same cluster is an effective way
4.0/).

Energies 2023, 16, 3187. https://doi.org/10.3390/en16073187 https://www.mdpi.com/journal/energies

Energies 2023, 16, 3187 2 of 21

to enhance the utilization of modern data center clusters [6,7]. However, it is not easy
to deploy different types of applications to increase server utilization without affecting
application performance. Their coexistence is a dilemma because it attempts to improve
resource utilization while the performance of online services declines as resource utilization
increases. Moreover, oversupply of peak power usage, fluctuating data traffic, and multi-
level power transmission infrastructure in large data centers can lead to serious power
budget fragmentation and inefficient power utilization [8].
Therefore, improving the energy efficiency of data centers has become one of the
main goals of data center construction and operation. Servers are the most essential
infrastructure of data centers, and their energy efficiency (EE) and energy proportionality
(EP) have become hot research topics in academia and industry. If the energy efficiency of
data center servers can be improved, the overall data center energy efficiency will also be
effectively improved.
Data center energy efficiency indicators play a very important role in data center
construction and operation management. Nowadays, there are a variety of evaluation
indicators for data center energy efficiency in the industry. Power efficiency (power us-
age effectiveness, PUE) is one of the most important indicators to evaluate data center
energy efficiency. PUE represents the ratio of total data center energy consumption to IT
equipment energy consumption. The total energy consumption of a data center includes
IT equipment energy consumption, cooling energy consumption, and lighting energy con-
sumption, among other things. The energy consumption of IT equipment includes server
energy consumption, network equipment energy consumption, and storage equipment
energy consumption.
This study is mainly based on the energy efficiency characteristics of the servers in the
data center. Improving the energy efficiency of all servers in the data center is also the top
priority for maintaining the high energy efficiency of the data center. The energy efficiency
of the data center is defined as Formula (1), where Ttotal represents the total number of
tasks that the data center can handle, and Ptotal represents the total power consumption of
all running servers in the data center:

Date Center Energy Efficiency = Ttotal /Ptotal . (1)

The energy efficiency of this data center is expressed as the number of load tasks per
watt and increasing the size of this metric means that when the number of tasks in the
data center is fixed, the data center PUE is reduced by reducing the power consumption of
IT devices.

1.2. Energy Efficiency and Energy Proportionality of Servers

The energy efficiency of data centers has become one of the main issues to be con-
sidered in the construction and management of data centers [9–14]. The most important
component of a data center is servers, as shown in Formula (2). The efficiency of a server is
defined as the ratio of server performance to power:

Server Energy Efficiency = Performance/Power. (2)

The higher the energy efficiency of a data center or server, the more tasks that can be
completed per watt of electricity. The energy efficiency indicator on a single server is often
used to describe the ratio of server performance to power consumption under a certain
utilization rate. Jiang [15] et al. found that the current peak energy efficiency of servers has
shifted from 100% utilization level to 70–80% utilization level, which shows that it is more
energy-saving to keep each server running within its energy efficiency peak range than at
100% utilization level. Figure 1 shows the energy efficiency of a server released in 2019
under different utilization rates. It can be seen that under different utilization rates, servers
exhibit different energy efficiency characteristics.
Energies 2023, 16, 3187 3 of 21
Energies 2023,
Energies 2023, 16,
16, 3187
3187 3 3of
of 21
21

18,000
18,000
16,000
16,000

Server energy efficiency(SSJ_OPS)

Server energy efficiency(SSJ_OPS)

14,000
14,000
12,000
12,000
10,000
10,000
8,000
8,000
6,000
6,000
4,000
4,000
2,000
2,000
0
0
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Server utilization(%)
Server utilization(%)

Figure 1. The relationship between a server utilization rate and energy efficiency released in 2019.
Figure 1. The relationship between a server utilization rate and energy efficiency
efficiency released
released in
in 2019.
2019.

In aIn
Indata center,
aa data
data thethe
center,
center, utilization
the utilization
utilizationraterate
of the
rate of servers
of the
the servers
serversis dynamic.
is
is dynamic.
dynamic. Simply
Simply
Simply comparing
comparing
comparing thethe
the
energy
energyefficiency
efficiency value
value of a
of server
a server under
under aa certain
certain utilization
utilization
energy efficiency value of a server under a certain utilization does not necessarily mean does
does not
not necessarily
necessarily mean
mean that
thatthat
theserver
the server
the is is
server energy
energy
is energy saving
saving inin
saving allall cases.
cases.
in all For
cases.For example,
example,
For if aif server
example, aifserver has
a serverhas high
high
has energy
energy
high effi-
efficiency
energy effi-
ciency
at at
70% 70% utilization,
utilization, it does it does
not not necessarily
necessarily mean mean
that that
this this
server
ciency at 70% utilization, it does not necessarily mean that this server has high energy server
has has
high high
energy energy
efficiency
efficiency
even under
efficiency eveneven under
low underlowlow
utilization.utilization.
In a dataIncenter
utilization. aIndata center
with
a data lowwith
center lowlow
utilization,
with utilization,
many many
servers
utilization, areservers
many idle. The
servers
areare
idle.
use of The
such use of
servers such
does servers
not does not
automatically automatically
improve the improve
energy
idle. The use of such servers does not automatically improve the energy efficiency of the energy
efficiency ofefficiency
the data of
center.
thethe
data
In center.
these
data cases,
center.In another
these
In thesecases, another
indicator
cases, is indicator
another needed
indicatortoisdetermine
needed
is needed to determine
the overallthe
to determine overall
energy
the energy
efficiency
overall energyof
efficiency
the of
servers. the servers.
This This
indicator indicator
is called is called
energy energy proportionality
proportionality
efficiency of the servers. This indicator is called energy proportionality (EP). Energy pro- (EP). (EP).
Energy Energy pro-
proportionality
portionality
refers to the
portionality refers to the
change
refers change
tointhe
server
change in server
energy energy
consumption
in server energy consumption with
with utilization
consumption withutilization
rate. It wasrate.
utilization It was
proposed
rate. by
It was
Rysckbosch
proposed by in 2007
Rysckbosch [16],in and
2007 its Formula
[16], and its(3) is as
Formula
proposed by Rysckbosch in 2007 [16], and its Formula (3) is as follows: follows:
(3) is as follows:
EPEP== 1=−11(𝐴𝑟𝑒𝑎
−−((𝐴𝑟𝑒𝑎𝑟𝑒𝑎𝑙 −−
Area 𝐴𝑟𝑒𝑎 𝑖𝑑𝑒𝑎𝑙 )/𝐴𝑟𝑒𝑎 𝑖𝑑𝑒𝑎𝑙 . . . (3) (3)
EP 𝑟𝑒𝑎𝑙 −Area
real 𝐴𝑟𝑒𝑎 ideal )/Area
𝑖𝑑𝑒𝑎𝑙 )/𝐴𝑟𝑒𝑎 ideal
𝑖𝑑𝑒𝑎𝑙
TheTheserver with
server withthethe most
most ideal energy
ideal energy consumption
consumption curvecurvehashas thethe
following
following charac-
charac-
The server with the most ideal energy consumption curve has the following character-
teristics: Assuming
teristics: Assuming that the
that power
the power consumption
consumption of the
of ideal
the server
ideal server is 100
is W
100 under
W under fullfull
istics: Assuming that the power consumption of the ideal server is 100 W under full load
load (100%
load (100% utilization),
utilization), the power
the power consumption
consumption should be reduced equally with the re- re-
(100% utilization), the power consumption shouldshould
be reduced be reduced
equallyequally
with the with the
reduction
duction
duction in the load,
in the and the power consumption at 80% load should be 80 W. WhenWhenisit is
it
in the load, andload, and the
the power power consumption
consumption at 80% load at should
80% load be should
80 W. When be 80itW. is completely
completely
completely idle, it
idle, should be 0 W. At this time, the energy proportional property of thisthis
idle, it should beit0 should
W. At this be 0time,
W. Atthethis
energytime,proportional
the energy proportional
property of this property
server of is one,
server
serveris one, which
one, which is the ideal
is the energy
ideal energy proportional
proportional characteristic.
characteristic. However,
However, in actual
in actual
which is the ideal energy proportional characteristic. However, in actual scenarios, the
scenarios,
scenarios, the server stillstill
needs power consumption when it isitcompletely idle, so the en-en-
server still the
needsserver
power needs
consumption power consumption
when when
it is completely is completely
idle, so the energy idle, so the
consumption
ergy consumption of the
ergy realreal
server is not proportional to thethe utilization rate. As Asshown
of theconsumption
real server is not of the server
proportional toisthe
not proportional
utilization rate. to As shown utilization
in Figure rate. shown
2, the Area real
in Figure
in Figure 2, the
2, Area
the real in Formula (3) represents the area between the real energy propor-
Area in Formula (3) represents the area between the real energy propor-
Formula (3) represents the area between the real energy proportional curve of the server
real
tional curve
tional curveof the server
of the server andandthetheabscissa;
abscissa;ideally,
ideally, Areaideal ideal
represents thethe
area between thethe
and the abscissa; ideally, Areaideal represents the areaAreabetween represents
the energy area between
proportional curve
energy
energy proportional
and theproportional curve
abscissa. Thecurve and
valueand the
of EP abscissa.
theranges
abscissa.The
from value
The value
0 to of EP ranges from
2. of EP ranges from 0 to 2. 0 to 2.

1
1
Normalized server power consumption

0.9
Normalized server power consumption

0.9
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0
0% 0 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Server utilization(%)
Server utilization(%)

Figure
Figure 2. Energy
2. Energy efficiency
efficiency curve
curve of the
of the server
server with
with an EP
an EP value
value of 0.98.
of 0.98.
Figure 2. Energy efficiency curve of the server with an EP value of 0.98.
Wong explained the meaning of EP on paper [17]: the EP value represents the change
Wong
Wong explained thethe
explained meaning of EP on on
paper [17]: thethe
EP EP
value represents thethe
change
in power consumption of ameaning of EP
server with paper
utilization; [17]:
the energy value represents
proportionality of datachange
center
in power
in consumption
powerhasconsumption of a server with utilization; the energy proportionality of data cen-
servers significantlyofimproved
a server with utilization;
in the the energy
past decade. Energyproportionality
proportionalityofhas
data cen-
a great
ter ter
servers has
servers significantly
has power improved
significantly improved in the past decade. Energy proportionality has a a
impact on the consumption of in the past
servers. In decade. Energy proportionality
their experiments, Barroso et al.has[18]
great impact on on
great thethe
power consumption of servers. In their experiments, Barroso et al.
et [18]
foundimpact
that servers power consumption
with energy of servers.
proportionality In their
closer to experiments,
one can saveBarroso
more power,al. [18]
so
Energies 2023, 16, 3187 4 of 21

they recommend that optimizing energy proportionality should be the primary goal of
server design. Ryckbosch et al. [16] studied the situation in which servers with higher EP
values can save power consumption. Under 10−50% utilization, servers with higher EP
values can ideally save 34% of energy consumption. Therefore, energy proportionality is of
great significance to server research and design. Improving the energy proportionality of
the server can greatly reduce the power consumption of the server. As the server energy
proportionality (EP) increases, data centers have an increasing demand for peak energy
efficiency-aware scheduling.

1.3. Data Center Simulation Tool

When designing a new data center, energy efficiency is one of the most concerning
issues for data center operators. In large-scale data centers, servers of different generations
are gradually installed year by year, which makes the data center heterogeneous in terms
of computing power and energy efficiency. Additionally, in existing data centers, internal
server distribution and infrastructure layouts that have been in operation for some time may
undergo significant changes, as will their original energy efficiency. Therefore, the extensive
energy efficiency of the data center depends to a large extent on the working range and
utilization level of each server. Although the current PUE (power usage efficiency) of large
data centers is already very low, if the server is not operating at full load working within
the maximum energy efficiency range, then the low value of PUE is meaningless. Therefore,
according to the workload characteristics, it is more energy-efficient to let different servers
operate within their most energy-efficient working ranges than to let all servers operate at
the same utilization level. This will greatly improve the energy efficiency of the data center.
In order to design a green data center with ideal PUE, many researchers have designed
various data center simulation tools. The simulated objects include data center servers,
network traffic, task placement, and heat maps, among others. The focus is on the power
consumption, communication, and application response time within the data center The
main problem of these simulation tools is poor stability and scalability.
Since workloads may change dynamically throughout the day, it is a challenge to
adaptively select a specific subset of servers to perform tasks while powering off the power
of the remaining servers so that the data center can operate with higher energy efficiency
as much as possible. Due to the paucity of energy efficiency simulators and incomplete
understanding of the overall energy efficiency of the data center, it is difficult for data center
operators to model the energy efficiency of the entire data center. Therefore, we designed
and implemented a data center energy efficiency simulator, namely Escope, which can
simulate and evaluate the energy efficiency of a data center.
The difference between the data center energy efficiency simulator designed in this
study and the above data center simulation is that it can simulate data center energy
efficiency. According to the different workload conditions and server configuration of the
data center, Escope explicitly models and visualizes the energy usage of the data center and
uses the energy efficiency characteristics of the servers in the data center to formulate work
server placement strategies to keep the servers at their most energy-efficient scope of work.
Through the simulation of data center energy efficiency, data center throughput, server
power consumption, power distribution, and other contents, the optimization algorithm
is used to simulate and evaluate the data center energy efficiency as a means to help
data center operators grasp the energy efficiency characteristics of each server and better
perform task scheduling.
Inspired by this, we have designed a new data center energy efficiency simulator
called Escope. We make the following contributions:
(1) Escope: This paper proposes the idea of simulating data center energy efficiency,
which is different from other simulators. Escope can clarify the energy usage of data
centers according to different workload conditions and server configurations and
model and visualize them;
Energies 2023, 16, 3187 5 of 21

(2) Flexibility: Escope allows for a variety of simulation strategies, and data center
operators can simulate energy efficiency according to their own needs;
(3) Depth: Through Escope’s simulation experiments, it has been shown that improving
energy efficiency in data centers is not simply a matter of running servers at their
highest energy efficiency point or running servers at full load. Escope’s simulation
algorithm can select the optimal set of servers and their corresponding utilization
levels to achieve optimal energy efficiency.
The organizational structure of this paper mainly consists of the following parts:
Section 1 explains the concepts related to data center energy efficiency. Section 2 introduces
the current relevant research content, while Section 3 expands on the key designs and
algorithms of Escope. Section 4 experiments with the Escope simulator and evaluates its
role. Finally, the discussion and summary are presented.

2. Related Studies
In order to design a green data center with ideal PUE, many researchers have designed
various data center simulation tools. Table 1 shows a comparative analysis of the different
simulation tools. Before 2009, distributed system simulators were less frequently used in
cloud computing environments, so Buyya et al. [19] and Calheiros et al. [20,21] proposed
CloudSim. This simulation software can produce seamless modeling and simulation of
cloud computing and their upper-layer application characteristics. CloudSim supports
simulation of cloud computing infrastructure and management services, so users can use
CloudSim to study specific system problems. CloudSim also introduced the simulation
of virtualized data centers and used NetworkCloudSim to expand the functions to better
support the simulation of internal communication in the data center [22]. However, the
scalability of CloudSim is poor, and experiments prove that the CloudSim simulator will
encounter various failures during the submission of the job.

Table 1. Comparison of different simulation tools.

Characteristics Advantages Disadvantages

Cloud computing
CloudSim Open source, user-friendly Poor scalability
environment simulation
Not integrated with the latest
Performance evaluation,
CloudAnalyst Visualization version of CloudSim
cost-benefit analysis
functionality
Resource allocation, load Simulates large-scale cloud
DartCSim+ Steep learning curve
balancing environments
Integrated with DVFS Considers only CPU power
DVFStoCloudSim Energy-aware simulation
technology consumption
Supports multiple workflow No dynamic resource
WorkflowSim Simulates workflow processes
models scheduling algorithm

In order to reduce or even eliminate the shortcomings and faults in CloudSim, many
researchers [23–30] improved the CloudSim simulator. For example, Li et al. [25] designed
the simulator DartCSim+, which supported power-aware network simulation. In order
to solve transmission failures caused by migration or network failures, DartCSim+ uses
a resubmission mechanism based on packet transmission. Bux et al. [26] solved the in-
homogeneity problem of CloudSim. On the basis of CloudSim, they added a process of
modeling instability in the cloud environment. It could simulate dynamic changes in
runtime performance and sudden changes during task execution issued by the failure.
In addition, Guérout and Monteil et al. [27] and others added a new patch of DVFS to
CloudSim, so that CloudSim could use DVFS to perform energy-aware simulation experi-
ments. Chen and Deelman et al. [28] et al. pointed out that ignoring the system failure and
overhead in the simulation workflow would have a significant impact on the simulation
Energies 2023, 16, 3187 6 of 21

experiment results, so they proposed WorkflowSim, which can be used to simulate the
workflow in a distributed environment. Wickremasinghe [29,30], also from the Buyya
team, proposed the visualization simulator CloudAnalyst, based on CloudSim. The main
purpose of the simulator was to achieve optimal scheduling of the data center under the
current configuration conditions. CloudAnalyst was designed directly based on CloudSim
and expanded some of CloudSim’s functions. It could be used to learn the behavior of
large-scale internet applications in the cloud computing environment and quickly conduct
simulation experiments. Kecskemeti [31] introduced a unified model of resource sharing
and a hierarchical energy monitoring framework, thus solving the scalability problem
of CloudSim.
Based on CloudSim, Tian et al. [32] developed the lightweight visual cloud computing
simulator CloudSched, which could support simulation modeling of large-scale cloud
computing applications. Using this simulator, users can customize their information,
data center information (number and location, etc.), resources, and other information
and simulate basic indicators such as data center response time and processing requests.
However, the simulator cannot simulate the amount of power consumption of the data
center. The MDCSim simulator [33] simulates the power consumption of the data center
and can model the characteristics of various devices (servers, switches, etc.) in the data
center. MDCSim can avoid building and processing similar simulation objects one by one,
so the required simulation time is significantly shortened, and the scalability is significantly
improved. CloudSim and MDCSim are event-based simulators. Their simulation accuracy
is insufficient. The MDCSim simulator is a commercial product, and its working principle
cannot be understood due to the lack of public source code. In order to improve the
simulation accuracy, Kliazovich [34] proposed a new simulator, GreenCloud, which is a
packet-level cloud data center simulator designed to evaluate the energy consumption cost
of data center operation. This simulator is an extension of the network simulator NS2 [35].
It mainly focuses on evaluating the power consumption of cloud communications and
provides a fine-grained power modeling and simulation tool for cloud data centers. Its key
advantage is that it fully supports the TCP/IP protocol model. For fine-grained simulation
of data center power consumption, DCWorms [36] provides simulations of data center
energy consumption, including energy consumption of cooling and ventilation systems
and energy consumption modeling of CPU, memory, and network in servers.
In order to support elastic cloud infrastructure simulation, Sriram [37] proposed the
SPECI simulation tool. It allows simulation of the performance and behavior of data centers
and simulates the functions and code of large data centers according to input size and
the middleware design strategy. SPECI consists of two packages: one for building data
center layouts and topologies and the other for executing experimental components, so it
has good scalability. Unlike the simulators introduced above, which are software-based
simulators, OpenCirrus [38,39] is an open cloud computing simulator based on software
and hardware, designed to support server design and management research in data centers.
The simulator has three main goals: to promote system-level research on cloud computing;
to encourage new cloud computing applications and application-level research; and to
provide experimental data sets to supply open API for cloud computing development.
The difference between the energy efficiency simulator designed in this research and
the above data center simulation is that Escope mainly simulates the energy efficiency
of the data center. By simulating data center energy efficiency, data center throughput,
server power consumption, and power distribution, optimized algorithms are used to
simulate and evaluate data center energy efficiency. It can also help data center operators
master the energy efficiency characteristics of each server, as a means to better perform
task scheduling.
Energies 2023, 16, 3187 7 of 21

Energies 2023, 16, 3187 7 of 21

3. Data Center Energy Efficiency Simulator—Escope Design
Experiments have shown that running servers in the best working range can improve
the energy
large efficiency
data centers, of servers.
hundreds Still, hundreds
of which of thousands
have different of servers
configurations andare deployed
models. Due into
large datahardware
different centers, hundreds of which
configurations, have different
the energy efficiencyconfigurations
characteristicsand models.
of servers areDue
also
to different
different hardware
[40,41]. configurations,
Therefore, data centerthe energyneed
operators efficiency characteristics
to understand of servers
the energy are
efficiency
also different [40,41]. Therefore, data center operators need to understand
and energy ratio of each server in order to effectively improve server utilization and rea- the energy
efficiency
sonable taskandscheduling
energy ratio of each
with server in of
the objective order to effectively
improving improve
the energy server of
efficiency utilization
the data
and reasonable task scheduling with the objective of improving the energy
center as much as possible. In order to enable data center operators to better understand efficiency of
the
the data
energycenter as much
efficiency as possible.ofInservers
characteristics order in
to the
enable
datadata center
center, operators
we have to better
developed the
understand the energy efficiency characteristics of servers in the data
data center energy efficiency simulator Escope. We use Escope to simulate the online num-center, we have
developed
ber of serversthe of
data center models
different energy efficiency
in the datasimulator
center and Escope. We use
the optimal Escope range
working to simulate
of the
the
servers to analyze the energy efficiency characteristics of the data center servers working
online number of servers of different models in the data center and the optimal and pro-
range of the
vide data servers
support fortodata
analyze
centerthe energy efficiency
operators. By modeling characteristics
data centerofenergy
the data center
efficiency,
servers and provide data support for data center operators. By modeling
task throughput, and power quota, optimization algorithms simulate and evaluate data data center
energy
center efficiency, task throughput,
energy efficiency and power
to help data center quota, optimization
operators understand algorithms
the energysimulate and
efficiency
evaluate data center
characteristics of eachenergy efficiency
server, to help data
thus improving taskcenter operators
scheduling andunderstand
data centerthe energy
energy ef-
efficiency characteristics of each server, thus improving task scheduling and data center
ficiency.
energy efficiency.
3.1. System
3.1. System Function
Function Design
Design
The architecture
The architecture of
of Escope
Escope isis shown
shown in
inFigure
Figure3.3. ItIt includes
includes components
components such
such as
as
crawler, simulator, database, selector, and web interface.
crawler, simulator, database, selector, and web interface.
Visual Interface
Web Interface (Web Interface)
Outside
Calculation Interface Calls
Selector EE Simulator Load Simulator

Storage Interface
Mysql Redis
External
Data
Data Interface
Monitor Benchmark Model VM Power Model
The
crawler
The Physical Layer interface

Server Server Server

Figure3.3. Escope
Figure Escopearchitecture.
architecture.

Escope
Escopeprovides
providesthe
thefollowing
followingfunctions:
functions:
(1)
(1) Escope
Escope integrates
integrates the
the monitoring
monitoring solution
solution for
for large-scale
large-scale distributed
distributed clusters
clusters inin the
the
open-source framework Zabbix, which can collect runtime data of all
open-source framework Zabbix, which can collect runtime data of all servers in the servers in the
cluster,
cluster, including
including temperature,
temperature, CPU CPU utilization,
utilization, memory
memory utilization,
utilization, server
server power
power
consumption, memory power consumption, and power
consumption, memory power consumption, and power management unit management unit (PMU)
(PMU) in-
information. Monitoring
formation. Monitoring items
items cancan be added
be added at anyat time
any time according
according to thetoneeds
the needs
of us-
of
ers;users;
(2)
(2) Escope provides aa load
Escope provides loadmodel,
model,which
whichintegrates
integratesaavariety
varietyofofcommonly
commonly used
used loads
loads in
in the industry. Users only need to enter information such as memory
the industry. Users only need to enter information such as memory utilization, disk utilization,
disk I/O,network
I/O, and and network I/O,Escope
I/O, and and Escope will automatically
will automatically completecomplete
the testthe
ontest
theon the
desig-
designated physical machine to obtain data information. The generated
nated physical machine to obtain data information. The generated data will support data will
support
the data the dataenergy
center centerefficiency
energy efficiency simulation;
simulation;
(3) For cloud computing operators, Escope provides a VM power consumption model,
(3) For cloud computing operators, Escope provides a VM power consumption model,
which can automatically estimate the VM power consumption based on the system
which can automatically estimate the VM power consumption based on the system
information collected on the server;
information collected on the server;
(4) In addition to automatically testing the energy efficiency of the server according to the
(4) In addition to automatically testing the energy efficiency of the server according to
load model, Escope can also automatically obtain the server configuration and energy
the load model, Escope can also automatically obtain the server configuration and
energy efficiency information (such as SPECpower) disclosed by each website to es-
tablish a server energy efficiency information database;
Energies 2023, 16, 3187 8 of 21

efficiency information (such as SPECpower) disclosed by each website to establish a

server energy efficiency information database;
(5) Users can make simulation policies and scenarios, such as setting the utilization rate of
all servers to no more than 30% or setting the utilization range for servers to simulate
the load of different business scenarios;
(6) Users can add new servers and server prices to the constructed data center. After
adding new machines, Escope can simulate the energy efficiency operation of the
newly constructed data center and calculate the cost–benefit based on the
electricity savings;
(7) Escope provides a web visual interface, so users can input various parameters and
display the simulation results. For example, it will classify and display the simulation
results according to server parameters (CPU model, memory size, release year, etc.)
to deeply analyze the energy efficiency distribution of data center servers.

3.2. Interface Design

The functions of each interface in Escope are as follows:
(1) Data interface: This interface is mainly used to test and collect energy efficiency
information of data center servers, including three components: monitoring, energy
efficiency testing, and VM power consumption model. The monitoring component is
used to collect information about the server. The energy efficiency testing component
integrates a variety of loads to perform energy efficiency testing on the server. The
VM power consumption model is used to estimate the power consumption of the VM
based on the monitoring information of the server;
(2) External data acquisition interface: This interface is responsible for acquiring server
energy efficiency information from the website that publishes server energy efficiency
information, sorting the data and storing it in the Escope database; hence, even
without real servers, Escope can still simulate data center energy efficiency;
(3) Storage interface: The function of the storage interface is to store and classify server
information obtained from various methods. Escope uses MySQL for persistent
storage and Redis for data caching, thus speeding up simulation efficiency while
ensuring data integrity;
(4) Calculation interface: This interface includes three components, namely, the selector,
the energy efficiency simulator, and the load generator. The selector can select quali-
fied server information and quantity from the database according to the parameters
provided by the user to construct the data center to be simulated. The load generator
can estimate the total throughput of the data center based on the typical load situation
of the data center and then input it into the simulator for simulation. The simulator
contains an energy efficiency simulation algorithm, which executes the simulation
algorithm according to the simulation strategy parameters input by the user. Then, it
generates a report after the simulation ends and displays the simulation result on the
Web interface;
(5) Visual interface: The visual interface has two functions. The first is to receive simu-
lation parameters provided by users, such as simulation strategies and server com-
bination strategies. The second is to display Escope simulation results for users
to analyze;
(6) External call: Escope provides an external call interface, and other systems in the
data center can call the simulation results generated by Escope through RPC (remote
procedure call).
The detailed execution process of Escope is shown in Figure 4. (The server energy
efficiency test process is omitted.)
(1) First, users collect information about the server’s energy efficiency, either by crawl-
ing from some web sites (SPECpower, for example) or by entering custom server
information through a web interface;
 Energies 2023, 16, 3187 9 of 21

(2) Then, users enter the selection parameters, the selector selects the server according
to the parameters and builds the data center to be simulated. The parameters can be
server type, server release year, server quantity, and CPU type;
(3) Next, the user inputs a simulation strategy, which can be to limit the total power
consumption of the data center or the throughput that the data center needs to achieve.
The goal of the simulator is always to select the server and its utilization rate under
the existing simulation strategy to maximize the energy efficiency of the data center,
thereby maximizing the throughput and energy efficiency of the data center;
(4) Finally, when the simulation is completed, the simulator will send the results to the
Energies 2023, 16, 3187 Web server in JSON format and generate a corresponding result report. The web
interface will display different types of simulation result data and charts according to
the simulation result, which is convenient for users to view and analyze the data.

Report

(4)Simulation results

(2)Simulation
strategy
Simulation Program
Data Acquisition Interface

(3)Data center
servers information (2)Selection
strategy
Selector

(1)Server (1)Server
information information

(1)Server
information

Crawler Program

website

Figure
Figure 4. Escope
4. Escope Simulation
Simulation Process. Process.
3.3. Energy Efficiency Simulation Algorithm for the Data Center
(1) The First, users
Escope collect
energy information
efficiency about the
simulation algorithm server’s
needs to solveenergy efficiency, ei
a combinatorial
ing from
optimization some
problem. web sites
Assuming that the (SPECpower,
data center has xfor example)
servers with differentor by entering cus
configura-
tions, the number of servers in each configuration is n, and there are N servers in total. Each
formation through a web interface;
server can run underutilization j (j = 1, 2, 3 . . . 10, representing 10% to 100% utilization).
(2) Then, users enter the selection parameters, the selector selects the ser
x j=10
to the parameters and max ∑ii=
builds=1 ∑the
j=1 qdata
ij ni ci , center to be simulated. The para

server type, server releases.t. ∑ii= x year,

j=10 server quantity, and CPU type;
=1 ∑ j=1 pij ni ci ≤ P, (4)
(3) Next, the user inputs a simulation j=10
strategy, which can be to limit th
s.t. ∑ j=1 ni ≤ n.
consumption of the data center or the throughput that the data ce
achieve.
The The goal of
power consumption ofserver
the simulator is always
i under the utilization ratetoofselect thethroughput
j is pij ; the server and its u
is qij , and the value
under of cj is either
the existing 0 or 1, which
simulation representstowhether
strategy maximize to turnthe on the server.efficien
energy
Formula (4) expresses the maximum throughput of the data center under the power limit,
center, thereby maximizing the throughput and energy efficiency of th
and Algorithm 1 shows the throughput maximization algorithm under the power limit.
(4) Finally, when the simulation is completed, the simulator will send the
Web server in JSON format and generate a corresponding result repor
terface will display different types of simulation result data and chart
the simulation result, which is convenient for users to view and analyz

3.3. Energy Efficiency Simulation Algorithm for the Data Center

The Escope energy efficiency simulation algorithm needs to solve a
Energies 2023, 16, 3187 10 of 21

Algorithm 1: Maximizing throughput simulation algorithm

Input: Data center server set N, Total power P
Output: Optimal solution bestvalue
1: for k in [1 , P] :
2: for i f rom [1, N + 1] :
3: if ni is f ailure :
4: bestvalue[i, j] = bestvalue[i − 1, j]
5: end if
6: for j f rom 1 to 10 :
7: if k < pij :
8: bestvalue[i, k] = bestvalue[i − 1, k]
9: else if bestvalue[i − 1, k] ≥ qij and bestvalue[i − 1, k] > bestvalue[i, k] :
10: bestvalue[i, j] = bestvalue[i − 1, k]
11: else : h i
12: bestvalue[i, k] = qij + bestvalue i − 1, k − pij
13: end if
14: end for
15: end for
16: bestValue = bestvalue[ N + 1, k + 1]

Algorithm 1 is similar to the multiple knapsack algorithm. The two-dimensional

array bestvalue [i, j] represents the maximum throughput of the first i server under power
consumption j, and bestValue represents the maximum throughput of the data center
under power consumption limit P. In this study, the power consumption limit P of the data
center is regarded as the backpack capacity, the power consumption of the server under the
utilization rate j as the weight, and the number of tasks that the server can handle under the
utilization rate j as the value. The function of the algorithm is to aggregate the appropriate
server and utilization and maximize the target value under restricted conditions. As shown
in Formula (5), Escope can also minimize the energy consumption of the data center while
limiting the total number of tasks in the data center.
i = x,j=10
min ∑i=1,j=1 pij ni ci ,
i = x,j=10
s.t. ∑i=1,j=1 qij ni ci ≥ T, (5)
j=10
s.t. ∑ j=1 ni ≤ n.

The data center power minimization algorithm is shown in Algorithm 2. The bestValue
in the algorithm represents the minimum power consumption required by the data center
when the data center throughput is T.
Based on Algorithms 1 and 2, users can calculate the optimal value under the data
center’s total power consumption or throughput limit. Assuming that the set of servers
selected by Algorithm 1 and Algorithm 2 is S, Algorithm 3 can output the optimal server
combination and the specific utilization rate of these servers according to bestvalue [i, j].
The two-dimensional array utl [i, j] represents the utilization rate selected by the server ni
under the limit j, and the utilization rate ranges from 10% to 100%. When bestvalue [i, L] is
greater than bestValue [I − 1, L], server ni will join the server set S, and the utilization rate
of server ni will be recorded at this time. When the simulated data center is too large, the
two-dimensional array bestvalue [i, j] may cause memory overflow. When j exceeds the
threshold (the size of the threshold is related to the size of the JVM), Escope will divide a
large knapsack problem into multiple small knapsack problems for simulation calculation,
and the segmentation accuracy will be lost (less than 1%). Although the accuracy is reduced,
it ensures that Escope can simulate data centers of any size. Users can freely adjust the
threshold according to the configuration of the computer running Escope, thus ensuring
the scalability of Escope.
Energies 2023, 16, 3187 11 of 21

Algorithm 2: Minimizing total power consumption simulation algorithm

Input: Data center server set N, Total throughput T
Output: Optimal solution bestValue
1: P = 0
2: for i in (1, N + 1) :
3: P = P + pi,10 // maximum power consumption of data center servers
4: for k in (1, P) :
5: for i in (1, N ) :
6: if ni is f ailure :
7: bestvalue[i, j] = bestvalue[i − 1, j]
8: for j f rom 1 to 10 :
9: if k < pij :
10: bestvalue[i, k] = bestvalue[i − 1.k]
11: else if bestvalue[i − 1, k] ≥ qij and bestvalue[i − 1, k] > bestvalue[i, k] :
12: bestvalue[i, j] = bestvalues[i − 1, k]
13: else: h i
14: bestvalue[i, k] = qij + bestvalue i − 1, k − pij
15: end if
16: end for
17: end for
18: while(bestvalue[i, k] > T ) :
19: k¯
20: end while
21: bestValue = bestvalues[ N + 1, k + 1]

Although the above algorithm can obtain the optimal solution of the combinatorial
optimization problem, its time complexity reaches O(n3 ), which requires more time to
simulate a large data center with a large number of servers. Therefore, this research also
integrates a simulated annealing algorithm (simulated annealing, SA) in Escope. The
simulated annealing algorithm is a method of seeking approximate solution optimization
problems based on a Monte Carlo design. The simulated annealing algorithm is essentially
a greedy algorithm. Because it adds random factors in searching for the optimal solution, it
has a certain probability to accept the sub-optimal solution, which may jump out of the
local optimal solution and reach the global optimal solution.

Algorithm 3: Server Selection Result output

Input: bestvalues[][]
L = Total throughput T || Total power P
Data center server set N
Output: Selected server set S
1: Initialize array utl [n + 1] [ L + 1]
2: for i in (n + 1, 1) :
3: if bestvalues[i ][ L] > bestvalues[i − 1][ L] :
4: S.add(ni−1 )
5: end if
6: if L is Power :
7: L = L − pi,utl [i−1][ L]
8: end if
9: if L is throughput :
10: L = L − qi,utl [i−1][ L]
11: end if
12: if L == 0 :
13: Break
14: end if
15: end for
Energies 2023, 16, 3187 12 of 21

As shown in Figure 5, assuming that the minimum point C is the optimal solution, the
simulated annealing algorithm will continue to move to the right with a certain probability
after searching for the local optimal solution B. By moving to the right, there is a certain
probability that B and C can be skipped. Therefore, the local minimum B is jumped out,
Energies 2023, 16, 3187 and the optimal value C is reached. The probability of accepting the sub-optimal solution
13 of 21
adopts the metropolis criterion. As shown in Equation (6), the probability that the particle
tends to balance at temperature T is exp (−∆E/(kT)), where E is the internal value at
temperature T, ∆E
of the solution is the variable,
(whether and K iscondition
the restriction the Boltzmann constant.
exceeds the threshold); and gradually
attenuate t (recalculate
( the probability of accepting the sub-optimal solution). Finally, the
1 E( xnew < Eold )
server combination generated

E( xnewat
)<the end
E( xold ) of the algorithm is the approximate optimal (6)
 so-
lution. exp − T E( xnew ≥ E( xold ))

FigureFigure 5. Simulated
5. Simulated annealing
annealing algorithmalgorithm
to find the to find the
optimal optimal
value valuepoint).
(the lowest (the lowest point).

The simulated
Algorithm 4. Simulatedannealing algorithm
Annealing is shown in Algorithm 4. When it is used to
algorithm.
solve the combinatorial optimization problem in this section, the internal energy E can
Input: Data center server set 𝑁, Total throughput 𝑇,
be assumed as the data center throughput, and the temperature T can be simulated as
Initial
the temperature
control parameter 𝐼𝑡 , Annealing
t to performrate the
𝑎𝑓 , simulated
number of annealing
balances 𝑏 algorithm. First, starting
from the initial i and
Number of iterations 𝑖𝑡𝑒𝑟 the initial value of the control parameter t, the current solution is
repeated to generate a new solution (delete or add a new server); calculate the objective
Output: Selected server set 𝑆
function difference (compared with the previous data center throughput); accept or discard
1： 𝒇𝒐𝒓of
iteration 𝑖 𝑖𝑛 (0,solution
the 𝑖𝑡𝑒𝑟) ∶ (whether the restriction condition exceeds the threshold); and
gradually
2： 𝒇𝒐𝒓 𝑗 𝑖𝑛 (0, 𝑏)t (recalculate
attenuate ∶ the probability of accepting the sub-optimal solution).
Finally,
3： the server combination
𝑛𝑜𝑤𝑉𝑎𝑙𝑢𝑒 = 𝑐𝑎𝑙𝑐𝑢𝑉𝑎𝑙𝑢𝑒() generated at the end of the algorithm is the approximate
optimal solution.
𝑠𝑒𝑟𝑣𝑒𝑟 =
4： The annealing rate𝑟𝑎𝑛𝑑𝑜𝑚()
in the simulated annealing algorithm has an undeniable impact on
5：efficiency
the 𝑆 of
= the
𝑝𝑢𝑡(𝑠𝑒𝑟𝑣𝑒𝑟)
algorithm. Although the temperature drops too fast to reach stability
quickly,
6： it𝒊𝒇will reduce the probability
𝑐𝑎𝑙𝑐𝑢𝑊𝑒𝑖𝑔ℎ𝑡() > 𝑃: of obtaining the optimal solution. If the temperature
drops too slowly, the algorithm will take too much time. In this study, the experimental
7： 𝑑𝑒𝑙𝑒𝑡𝑒(𝑠𝑒𝑟𝑣𝑒𝑟)
annealing rate is 0.95. The more you set the number of balances, the fewer iterations
8： need, but
you 𝑐𝑜𝑛𝑡𝑖𝑛𝑢𝑒
the time for a single iteration becomes longer. The setting of the initial
temperature
9： e𝒏𝒅will
𝒊𝒇 affect the search range of the solution. The higher the temperature, the
higher the quality of the final solution, but the algorithm will take longer. Compared
10： 𝒊𝒇 𝑐𝑎𝑙𝑐𝑢𝑉𝑎𝑙𝑢𝑒() >= 𝑛𝑜𝑤𝑉𝑎𝑙𝑢𝑒:
with the knapsack algorithm, the simulated annealing algorithm can obtain sub-optimal
11：
solutions, 𝑐𝑜𝑛𝑡𝑖𝑛𝑢𝑒
but the time complexity is reduced to O (2x(log (n))2 ). We use two different
simulation
12： 𝒆𝒍𝒔𝒆:
algorithms in Escope to minimize power consumption for 10,000, 100,000, and
1 million servers. The input is the number of tasks that need to be processed. Algorithm 2 is
𝑐𝑎𝑐𝑙𝑢𝑉𝑎𝑙𝑢𝑒−𝑛𝑜𝑤𝑉𝑎𝑙𝑢𝑒
𝑀𝑎𝑡ℎ. 𝑟𝑎𝑛𝑑𝑜𝑚()
13： as Algorithm
labeled < 𝑀𝑎𝑡ℎ. 𝑒𝑥𝑝(
1, and Algorithm 4 Labeled as𝐼𝑡Algorithm ) 2. The experimental results
are shown in Table 2.
14： 𝒆𝒏𝒅 𝒊𝒇
15： 𝒆𝒏𝒅 𝒇𝒐𝒓
16： 𝐼𝑡 = 𝐼𝑡 ∗ 𝑎𝑓
17： 𝒆𝒏𝒅 𝒇𝒐𝒓

The annealing rate in the simulated annealing algorithm has an undeniable impact
Energies 2023, 16, 3187 13 of 21

Algorithm 4: Simulated Annealing algorithm

Input: Data center server set N, Total throughput T,
Initial temperature It , Annealing rate a f , number of balances b
Number of iterations iter
Output: Selected server set S
1: for i in (0, iter ) :
2: for j in (0, b) :
3: nowValue = calcuValue()
4: server = random()
5: S = put(server )
6: if calcuWeight() > P :
7: delete(server )
8: continue
9: end if
10: if calcuValue() >= nowValue :
11: continue
12: else :  
13: Math.random() < Math.exp cacluValueI−t nowValue
14: end if
15: end for
16: It = It ∗ a f
17: end for

Table 2. Simulation time comparison of different algorithms.

Number of Servers Number of Servers Number of Servers

(104 ) (105 ) (105 )
Algorithm 1
54 s 228 s 1498 s
completion time
Algorithm 1 accuracy 100% 100% 99.4%
Algorithm 2
8s 19 s 488 s
completion time
Algorithm 2 accuracy 99.1% 97.6% 94.1%

Algorithm 1 has a smaller accuracy in the case of one million units. The reason is that
there are too many servers to be simulated, which requires decomposition and processing,
resulting in a decrease in simulation accuracy. On the other hand, Algorithm 2 has a faster
calculation time. In the case of a million scale, the speed is 68% higher than Algorithm 1,
but it also loses 6% accuracy.
Data center operators can choose different simulation algorithms according to their
needs. If a fast simulation is required and the optimal value is not required, then the
simulated annealing algorithm can be selected. If the optimal solution is needed, the
multiple knapsack algorithm can be used for simulation.

4. Experiment and Analysis

In order to verify the effectiveness of energy efficiency simulator Escope, four kinds
of simulation data centers were established in this section. The data center server was
constructed with the server energy efficiency information released by SPECpower in
2017–2019; 135 types of servers were selected, each type of the server was set to 50, and the
total number of servers in a data center was 6750. We set two different simulation goals.
One goal was to calculate the number of servers online when the data center throughput
was maximized and the utilization group located under the power consumption limit of the
data center. The second goal was to calculate the online number of servers that minimize
the power consumption of the data center and the utilization rate when the task load of the
data center was determined. Escope would simulate the optimal solution for the highest
Energies 2023, 16, 3187 14 of 21

energy efficiency in the data center by simulating which servers were turned on and at
which utilization rate among the 6750 servers. The overview of the four different types of
data centers is as follows:
Energies 2023, 16, 3187 15 of 21
(1) Data center #1: the server can choose to run at any utilization;
(2) Data center #2: the server is always running at the highest energy efficiency;
(3) Data center #3: server utilization is less than 30%;
(4) In the simulation
Data center #4: theof this
serverarticle,
runsdata center
at 100% #1 has no restrictions on server utilization,
utilization.
and all servers can choose to operate at any utilization. Intuitively, as long as each server
In the simulation of this article, data center #1 has no restrictions on server utilization,
runs on servers
and all the highest
can EE, the data
choose centerat
to operate can achieve
any the highest
utilization. energyasefficiency,
Intuitively, long as eachso we set
server
up
runs on the highest EE, the data center can achieve the highest energy efficiency, so we by
data center #2 to verify whether this idea is correct. The server portfolio simulated set
data center #2 will all run at its highest EE utilization. Data center #3 simulates
up data center #2 to verify whether this idea is correct. The server portfolio simulated by a typical
data
datacenter
centersituation
#2 will allwhere
run atthe itsserver
highestusage rate is less Data
EE utilization. than 30%.
centerThis situation wastes
#3 simulates a typicala
lot of resources but is not completely useless. The servers selected in data
data center situation where the server usage rate is less than 30%. This situation wastes a center #4 will
run at resources
lot of 100% utilization,
but is notand the purpose
completely of theThe
useless. setting is toselected
servers verify whether running
in data center the
#4 will
server at 100% utilization is an optimal policy. No matter which simulation
run at 100% utilization, and the purpose of the setting is to verify whether running the strategy is
used,
serverEscope
at 100%will select the
utilization is anbest combination
optimal policy. No of matter
serverswhich
under the current
simulation situation
strategy to
is used,
maximize
Escope will theselect
energy
theefficiency of the data
best combination center while
of servers undersatisfying
the currentthe current to
situation strategy.
maximize
the energy efficiency of the data center while satisfying the current strategy.
4.1. Simulation of Maximum Throughput in Data Center with Limited Power Consumption
4.1. Simulation
Constrainedof Maximum Throughput
by data center power in Data Center with
infrastructure and Limited
cooling Power Consumption
conditions, data center
servers must operate
Constrained by under strict power
data center power infrastructure
restrictions. When the power
and cooling of all the data
conditions, servers on
center
servers
the rack must
exceeds operate under
the rated strictlimit,
upper power therestrictions.
servers willWhen
power the power
out, whichof will
all the servers
affect the
on theoperation
stable rack exceeds
of thethe
datarated upper
center. limit, the
Therefore, theservers
limitedwill power
power out, which
consumption will prob-
poses affect
the stable operation of the data center. Therefore, the limited power consumption
lems for data center operators: When the total power of the data center is limited, which poses
problems
types for data
of servers center
should dataoperators: When the
center operators total power
choose, and how of the
many data centershould
servers is limited,
run
which
in eachtypes
model? of servers shouldatdata
Furthermore, what center operators
utilization choose,
range shouldandthese
howservers
many servers should
keep maxim-
run in
izing theeach model? of
throughput Furthermore,
the data center?at what utilization range should these servers keep
maximizing the throughput of the data center?
In this section, several simulation experiments will be conducted on the four afore-
In this section, several simulation
mentioned data centers to explore the energy experiments
efficiencywill be conducted
operating range ofondifferent
the four types
afore-
mentioned data centers to explore the energy efficiency operating range
of servers in data centers under power constraints. The upper limit of the total power of different types
of
of servers in data centers under power constraints. The upper limit of the
each data center in the experiment is 100 KW to 1000 KW, and the increased step length is total power of
each
10 KW. data
Thecenter in the experiment
experimental results areisshown
100 KWintoFigures
1000 KW, and
6 and 7.the increased step length is
10 KW. The experimental results are shown in Figures 6 and 7.
#1 #2
3000

#3 #4
2500
Number of servers online

2000

1500

1000

500

0
100 200 300 400 500 600 700 800 900 1000
Power consumption cap(Kw)

Figure
Figure 6.
6. Number
Numberof
ofservers
serversonline
online for
for different
different data
data centers.
centers.

The simulation results of data center #1 and data center #2 are similar, but the through-
2.E+10

put1.E+10
of data center #1 #1 is always
#2 greater than that of data center #2. The reason for this
result
1.E+10
is that although the simulation strategy of data center #2 always selects the server
#3 #4
utilization at the peak EE, in some cases, this choice does not maximize the total throughput.
1.E+10
Throughput

For example, when the available power consumption is 200 W, the selected server running
at 8.E+09
100% utilization needs 200 W, and the server running at EE peak (assuming 80%) needs
1806.E+09
W. Therefore, #1 will select 100% utilization, while # 2 will choose 80% utilization,
4.E+09

2.E+09

0.E+00
100 200 300 400 500 600 700 800 900 1000
Power consumption(Kw)
 Number of servers online
2000

1500

1000
Energies 2023, 16, 3187 15 of 21
500

0
100 200 300 400 500 600 700 800 900 1000
so the remaining 20Power
W consumption
will be cap(Kw)
wasted. Obviously, since data center #1 can choose any
utilization rate and the simulated optimal combination of servers can maximize throughput,
Figure 6. Number of servers online for different data centers.
the throughput of data center #1 is always the largest.
2.E+10
#1 #2
1.E+10

1.E+10 #3 #4
1.E+10
Throughput

8.E+09

6.E+09

4.E+09

2.E+09

0.E+00
100 200 300 400 500 600 700 800 900 1000
Power consumption(Kw)

Figure 7. Throughput of different data centers.

The throughput of data center # 3 is much lower than that of other data centers,
indicating that traditional data centers waste resources when server utilization is less than
30%, thus reducing the throughput and energy efficiency of the entire data center. For
data center # 4, the number of servers online is always the lowest, because server power
consumption is always the highest at 100% utilization, so it is easy to reach the power limit.
This also verifies that the strategy of unplanned running servers at the highest utilization
does not improve the energy efficiency of the data center.
By analyzing the experimental results, the following conclusions can be drawn:
(1) Under the same power consumption limit, the number of online servers at peak energy
efficiency (data center #2) is about 29.86% higher than servers at 100% utilization (data
center #4), and the total number of tasks in the entire data center (ssj_ops) increased
by 7.17%;
(2) The number of servers in data center #1 is slightly lower than that in data center #2,
but the throughput is 2% higher than that of servers running at peak energy efficiency
utilization (data center #2).
(3) The server utilization rate of the traditional data center is lower than 30% (data center
#3), and the average throughput is 33% lower than that of data center #1. However,
the number of online servers has increased by 25% when compared to #1. The largest
number of online servers means that data center #3 has better redundancy, which can
ensure that the data center provides stable services.
Judging from the energy efficiency distribution of Escope’s selection of server col-
lections, the energy efficiency of servers released in recent years has been significantly
improved when compared to many years ago, and the average EE and EP have been greatly
improved. Table 3 shows the average EE and EP of different data centers when the power
consumption is limited to 1000 KW. Comparing the average EE values of data center #1 and
data center #2, it can be seen that it is a better strategy to choose a server with a high EE
value, but this is not always optimal. The largest EP is chosen in data center #2, because EP
represents the variation in server power consumption with utilization. Therefore, servers
with large EE may not have the largest EP, but servers with larger EPs tend to have a higher
EE value when working at low utilization (10−30%).
 Energies 2023, 16, 3187 16 of 21

Table 3. The average EE and EP of the four data centers with a power limit of 1000 KW.

Data Center Average EE Average EP Average Utilization

#1 15,331 0.86 79.75%
#2 15,148 0.87 77.95%
Energies 2023, 16, 3187
#3 10,571 0.92 30%
#4 14,327 0.83 100%

center.
4.2. In thisof section,
Simulation the simulation
Power Minimization goal
with Data of all
Center four data
Throughput centers is to minimize dat
Preserved
power When consumption while maximizing
the size and business of the data center dataare center
stable,throughput.
the throughput Inof
the
thesimulation,
data t
center remainsof
throughput stable.
eachMinimizing
data centertheispower set from consumption
1 × 108 toof1the × 10 data centerawhen
10, with step the
size of 1 ×
throughput is fixed is also a key issue for improving the energy efficiency of the data
The simulation results are shown in Figures 8 and 9. For data center #4, the
center. In this section, the simulation goal of all four data centers is to minimize data center
of online
power servers while
consumption is always the lowest.
maximizing data centerThis is because
throughput. In running at 100%
the simulation, utilization
the total
server means
throughput of eachthatdatathecenter
server needs
is set from 1to ×consume
8
10 to 1 × 10 the, most
10 with a power
step sizeand
of 1 ×can
108handle
. t
tasks. However, maximum peak processing capacity does not imply highest ener
The simulation results are shown in Figures 8 and 9. For data center #4, the number of
Energies 2023, 16, 3187 online servers is alwaysdata the lowest. This 17 of 21
ciency. Therefore, center #4is has
because
an running
average at 100%
power utilization on the server
consumption increase
means that the server needs to consume the most power and can handle the most tasks.
(about 30 KW) when compared to data center #2, with the highest energy efficien
However, maximum peak processing capacity does not imply highest energy efficiency.
center. In this section, the simulation goal of all four data centers is to minimize data center
zation.
Therefore,
power
The data
consumption
average
center
while
energy
#4
maximizing anconsumption
has data average powerof
center throughput.
data centerincrease
consumption
In the simulation, the
#1 is reduced
total of 1.7% by 0.85% wh
(about
pared
30 KW) to
throughput ofdata
when center
eachcompared
data center #2. For
istosetdata the
1 × 10traditional
fromcenter #2,
8 to 1 ×with thedata
1010, with ahighest center
step size energy
of #3,
1 × 10 due to the
8.efficiency low resource
utilization.
The The simulation
average energyresults are shown in Figures
consumption of data 8 and 9. For
center #1 data
is center #4, by
reduced the 0.85%
numberwhen compared to
tion
of online
rate, the
servers
averagelowest.
is always
power consumption increased byon36% (about 200 KW) whe
data center #2. For thethetraditional This data
is because
centerrunning at 100%
#3, due utilization
to the low resourcethe utilization rate,
pared
server to the
means other
that the serverthree
needs to high utilization
consume the most power data andcenters.
can handleHowever,
the most the number of onli
the average power consumption increased by 36% (about
tasks. However, maximum peak processing capacity does not imply highest energy effi-
200 KW) when compared to the
ers in data
other Therefore,
ciency. three high center #3
utilization
data
has
center #4 has dataincreased
an centers.
average power
by an
However, average
the number
consumption
of 45%
increase of online servers in data to oth
of 1.7%
when compared
centers,
center30#3
(about KW) which
haswhen can
increased
compared better
bytoan guarantee
average
data ofwith
center #2, the
45%thewhen quality
highest compared of service.
to other
energy efficiency data centers, which
utili-
zation. The average
can better energythe
guarantee consumption
quality of of data center #1 is reduced by 0.85% when com-
service.
pared to data center #2. For the traditional data center #3, due to the low resource utiliza-
tion 3000
rate, the average power consumption increased by 36% (about 200 KW) when com-
#1 #2 #3 #4
pared to the other three high utilization data centers. However, the number of online serv-
ers in data
2500 center #3 has increased by an average of 45% when compared to other data
centers, which can better guarantee the quality of service.
Number of server online

2000
3000
#1 #2 #3 #4

1500
2500
Number of server online

2000
1000
1500

1000 500

500
0
0
1.E+08 1.E+09 2.E+09 3.E+09 5.E+09 6.E+09 7.E+09 8.E+09 9.E+09 1.E+10
1.E+08 1.E+09 2.E+09 3.E+09 5.E+09 6.E+09 7.E+09 8.E+09 9.E+09 1.E+10
Throughput Throughput

Figure 8. Number of servers online in different data centers.

Figure8.8.Number
Figure Numberof servers onlineonline
of servers in different data centers.
in different data centers.
1,200,000
#1 #2 #3 #4
Datacerner consumption(W)

1,000,000
1,200,000
800,000 #1 #2 #3 #4
Datacerner consumption(W)

1,000,000
600,000

400,000
800,000
200,000

0 600,000
1.E+08 1.E+09 2.E+09 3.E+09 5.E+09 6.E+09 7.E+09 8.E+09 9.E+09 1.E+10
Throughput
400,000
Figure
Figure 9. 9.
Total server
Total powerpower
server consumption of differentof
consumption data centers.
different data centers.
200,000
Table 4 shows the release year of the server selected by data center #1 in the simula-
0
tion. It can be seen from the table that in order to achieve maximum energy efficiency, the
1.E+08 1.E+09 2.E+09 3.E+09 5.E+09 6.E+09 7.E+09 8.E+09 9.E+09 1.E+10
choice of the simulator is related to the server energy efficiency characteristics but not the
Throughput
release year, which shows that Escope can choose the best server combination according
to different strategies.
Figure 9. Total server power consumption of different data centers.
Table 4. Release year and number of servers under 1 × 1010 throughput in data center #4.
Energies 2023, 16, 3187 17 of 21

Table 4 shows the release year of the server selected by data center #1 in the simulation. It
can be seen from the table that in order to achieve maximum energy efficiency, the choice of the
simulator is related to the server energy efficiency characteristics but not the release year, which
shows that Escope can choose the best server combination according to different strategies.

Table 4. Release year and number of servers under 1 × 1010 throughput in data center #4.

Server Name Price (USD) EE Number(U)

Fusion 2288 H V5 7768 13,478 10
Fujitsu RX2540 M4 11,172 12,842 10
Fujitsu RX4770 M4 9270 12,828 10
Energies 2023, 16, 3187 18 of 21
ThinkSystem SR950 10,025 12,377 10
Sugon I820-G30 9487 12,306 10

Escope will use load generators to generate tasks that the data center needs to com-
plete.Escope willgeneration
The load use load generators to generate
result is shown tasks that
in Figure the data
10, which centerthe
shows needs to complete.
throughput of
The load generation result is shown in Figure 10, which shows the throughput of
the data center in a day. The maximum task volume of the data center is 6.0 × 10 , and the8 the data
center load
hourly in a day. The
in the daymaximum
is set to atask
stepvolume of theon
shape based data center is 6.0
SPECpower × 108 , and the hourly
modeling.
load in the day is set to a step shape based on SPECpower modeling.
6.00E+08

5.00E+08

4.00E+08
Throughput

3.00E+08

2.00E+08

1.00E+08

0.00E+00
1 2 3 4 5 6 7 8 9 10 12 14 16 18 20 22 24
Time(hour)

Figure10.
Figure Throughputof
10.Throughput ofthe
thedata
datacenter
centerper
perday.
day.

The simulation
The simulationresults
resultsare
are shown
shown inin Tables
Tables 55 and
and 6.
6. The
Theenergy
energyefficiency
efficiencyof
of data
data
center#5
center #5isis7297,
7297,and
andthe
theenergy
energyefficiency
efficiencyof
ofdata
datacenter
center#6
#6isis13758.
13758.Based
Basedononthe
theaverage
average
electricity price of 1 CNY/kWh in China in 2019, small data center #6 compares
electricity price of 1 CNY/kWh in China in 2019, small data center #6 compares with data with
data center #5 by saving about CNY 1000 per day. Using the Escope simulation,
center #5 by saving about CNY 1000 per day. Using the Escope simulation, it takes 3418 it takes
3418 days to recover the cost of purchasing a new server by saving power.
days to recover the cost of purchasing a new server by saving power.
Table 5.
Table Simulation results
5. Simulation results of
of data
data center
center #5.
#5.

Throughput Average Number of Severs

ThroughputAverage Consumption (W) Number of Severs Online Time
Time (hour)
(Hour)
Consumption (W) Online
𝟏𝐄𝟖 11,660 36 2
𝟐𝐄𝟖 1E8 11,660
23,645 36 47 2 2

𝟑𝐄𝟖 2E8 23,645

36,967 47 71 2 2
𝟒𝐄𝟖 3E8 52,873
36,967 71 140 2 2
𝟓𝐄𝟖 4E8 70,823
52,873 140162 2 2
𝟔𝐄𝟖 5E8 91,594
70,823 162 210 2 14
6E8 91,594 210 14
Table 6. Simulation results of data center #6.

Throughput Average Consumption (W) Number of Severs Online Time (hour)

𝟏𝐄𝟖 6458 22 2
𝟐𝐄𝟖 13,082 37 2
𝟑𝐄𝟖 19,946 41 2
𝟒𝐄𝟖 26,930 50 2
𝟓𝐄𝟖 37,193 67 2
𝟔𝐄𝟖 49,052 38 14
Energies 2023, 16, 3187 18 of 21

Table 6. Simulation results of data center #6.

Average Number of Severs

Throughput Time (Hour)
Consumption (W) Online
1E8 6458 22 2
2E8 13,082 37 2
3E8 19,946 41 2
4E8 26,930 50 2
5E8 37,193 67 2
6E8 49,052 38 14

The larger the size of the data center, the greater the electricity consumption, the higher
the electricity price, and the faster the cost recovery will be. However, this number is only
a reference value for data center operators. Data center operators can simulate data center
energy efficiency by configuring different server combinations so as to choose the most
suitable server for their data center and reduce procurement costs.

5. Discussion
In practice, on the one hand, data center servers must run under strict power restric-
tions, and operators need to consider which types of servers should be selected and how
many servers should be run in each model; they also need to understand what utilization
range must be maintained to maximize the throughput of the data center. On the other
hand, when the size of the data center and the business is stable, the throughput of the
data center will remain stable. Minimizing power consumption in the data center when
throughput is fixed must also be considered. Escope can simulate the energy efficiency of
the data center. By entering the power consumption limit of the data center, we can simulate
the maximum throughput of the data center under this power limit. Simulations can also
be run to determine which servers should be started and what utilization range the servers
should be run in as a means to achieve the maximum throughput. At the same time, by
entering the number of tasks that the data center needs to handle, Escope can calculate the
minimum power consumption required by the data center server to handle these tasks. In
terms of specific algorithm selection, data center operators can choose different simulation
algorithms according to their needs. If a fast simulation is required and the optimal value
is not necessary, then the simulated annealing algorithm can be selected. If the optimal
solution is needed, multiple simulation algorithms can be selected. The backpack algorithm
is simulated.

6. Conclusions
This study starts with the reduction in data center power consumption; then, related
research on data center energy efficiency and energy proportionality are introduced. Subse-
quently, we develop the design and experimental analysis of the energy efficiency simulator
Escope, and now we can draw the following conclusions:
(1) The energy efficiency of the data center cannot be improved by running the server
at the highest energy efficiency point or by running the server under full load. The
simulation algorithm provided by Escope can select the optimal server set and their
corresponding utilization rate;
(2) Escope can set a variety of simulation strategies, and data center operators can simu-
late data center energy efficiency according to their own needs;
(3) When limiting the server utilization rate to less than 30%, almost all simulation
results of the server run at 30% utilization rate. This is because under the existing
architecture, the energy efficiency of the server at 30% utilization rate must be higher
than a utilization rate of less than 30%, so most of the selected servers run at 30%
utilization rate;
Energies 2023, 16, 3187 19 of 21

(4) Escope can calculate the electricity cost saved by introducing new servers in the
data center. This function provides an important reference for operators to purchase
servers and design data centers.
In addition to the data centers constructed in the study, operators can also build data
centers that conform to the actual situation. According to different management purposes,
the data center may have different operation strategies. Escope can simulate the energy
efficiency of the data center according to different strategies, help data center operators
understand the energy efficiency characteristics of the data center, and require the data
center task scheduler to make the best decision.
In future work, we plan to set up server performance data for more benchmark types
such as hybrid web server benchmarks and memory intensive benchmarks. Further, we
hope to allow users to customize the benchmark. We will add a monitoring system to
Escope, allowing Escope to automatically monitor server performance. Administrators
can test more benchmarks (rather than SPECpower only), and Escope can automatically
generate server performance data to bring the simulation closer to reality.
In addition, due to the fact that new applications of artificial intelligence require a large
amount of computing and storage resources to support their algorithms and models, these
resources need to be supported and maintained in data centers, resulting in significant en-
ergy consumption. In the future, we can further use CPU and GPU to accelerate simulation
calculations in data centers and improve simulation speed and accuracy. Quantum com-
puting, on the other hand, can better handle complex problems such as optimization and
derivation of artificial intelligence algorithms, thereby improving the level of intelligence
in data centers. By combining these tools, different data center scenarios can be simulated,
and future performance and energy costs can be predicted.

Author Contributions: Conceptualization, C.J. and C.C.; methodology, C.J. and L.Y.; software, C.Y.
and Y.Q.; validation, J.L., Y.L. (Yan Li) and Y.L. (Yang Li); formal analysis, Y.L. (Yan Li) and Y.L. (Yang
Li); resources, L.Y.; data curation, C.Y.; writing—original draft preparation, C.J.; writing—review and
editing, L.Y.; visualization, J.L.; supervision, C.J.; project administration, L.Y.; funding acquisition, C.J.
All authors have read and agreed to the published version of the manuscript.
Funding: This research was funded by the Science and Technology Project of State Grid Corporation
of China (No. 52993920002M), and the Natural Science Foundation of China (No. 61972118). The
funder was not involved in the collection, analysis, or interpretation of the data or manuscript.
Data Availability Statement: All data from the study are presented in the manuscript.
Acknowledgments: The authors thank all participants for their time and cooperation throughout
the study.
Conflicts of Interest: The authors declare no conflict of interest.

References
1. Cook, G. How Dirty Is Your Data: A Look at the Energy Choices that Power Cloud Computing; Greenpeace: Amsterdam, The
Netherlands, 2011.
2. Pettey, C. Gartner Estimates ICT Industry Accounts for 2 Percent of Global CO2 Emissions; Gartner Symposium/ITxpo: San Francisco,
CA, USA, 2007. Available online: http://www.gartner.com/us/symposiumwest (accessed on 3 January 2023).
3. Jyothi, S.A.; Curino, C.; Menache, I. Morpheus: Towards Automated SLOs for Enterprise Clusters. In Proceedings of the USENIX
OSDI 2016, Savannah, GA, USA, 2–4 November 2016; USENIX: Berkeley, CA, USA, 2016; pp. 117–134.
4. Rajan, K.; Kakadia, D.; Curino, C. PerfOrator: Eloquent performance models for Resource Optimization. In Proceedings of the
Seventh ACM Symposium on Cloud Computing, Santa Clara, CA, USA, 5–7 October 2016; pp. 415–427.
5. Xu, G.; Xu, C.Z. Prometheus: Online estimation of optimal memory demands for workers in in-memory distributed computation.
In Proceedings of the 2017 Symposium on Cloud Computing, Santa Clara, CA, USA, 24–27 September 2017; p. 655.
6. Yan, Y.; Gao, Y.; Chen, Y.Y. Tr-spark: Transient computing for big data analytics. In Proceedings of the Seventh ACM Symposium
on Cloud Computing, Santa Clara, CA, USA, 5–7 October 2016; pp. 484–496.
7. Chen, W.; Rao, J.; Zhou, X. Preemptive, low latency datacenter scheduling via lightweight virtualization. In Proceedings of the
USENIX Annual Technical Conference (USENIX ATC 17), Santa Clara, CA, USA, 12–14 July 2017; pp. 251–263.
Energies 2023, 16, 3187 20 of 21

8. Hsu, C.-H.; Deng, Q.; Mars, J. SmoothOperator: Reducing Power Fragmentation and Improving Power Utilization in Large-scale
Datacenters. In Proceedings of the 23rd International Conference on Architectural Support for Programming Languages and
Operating Systems(ASPLOS ‘18), Providence, RI, USA, 13–17 April 2018; pp. 535–548.
9. Gu, L.; Zhou, F.; Meng, H. Study on energy consumption and energy efficiency of data centers in China. China Energy 2010, 32,
42–45.
10. Tantar, A.A.; Tantar, E. A survey on sustainability in ICT: A computing perspective. In Proceedings of the Companion Publication
of the 2014 Annual Conference on Genetic and Evolutionary Computation, Vancouver, BC, Canada, 12–16 July 2014; pp.
1213–1220.
11. Zhang, W.; Wen, Y.; Wong, Y.W.; Toh, K.C.; Chen, C.H. Towards joint optimization over ICT and cooling systems in data centre: A
survey. IEEE Commun. Surv. Tutor. 2016, 18, 1596–1616. [CrossRef]
12. Xing, B.; Lu, M.; Jin, P.; Huang, G.; Yue, L. Energy efficient query Optimization Technology for Green data Center. Comput. Res.
Dev. 2019, 56, 1821–1831.
13. Song, J.; Sun, Z.; Liu, H.; Bao, Y.; Yu, Y. Research Progress on Energy consumption Optimization of Hybrid Power supply data
Center. J. Comput. Sci. 2018, 41, 36–54.
14. Shen, H. Energy consumption calculation and energy efficiency analysis in data center energy conservation evaluation report.
Eng. Res. 2019, 4, 245–246.
15. Jiang, C.; Wang, Y.; Ou, D.; Qiu, Y.; Li, Y.; Wan, J.; Luo, B.; Shi, W.; Cerin, C. EASE: Energy Efficiency and Proportionality
Aware Virtual Machine Scheduling. In Proceedings of the 30th International Symposium on Computer Architecture and High
Performance Computing (SBAC-PAD2018), Paris, France, 24–27 September 2018.
16. Ryckbosch, F.; Polfliet, S.; Eeckhout, L. Trends in server energy proportionality. Computer 2011, 44, 69–72. [CrossRef]
17. Wong, D.; Annavaram, M. Knightshift: Scaling the energy proportionality wall through server-level heterogeneity. In Proceedings
of the 2012 45th Annual IEEE/ACM International Symposium on Microarchitecture, Vancouver, BC, Canada, 1–5 December 2012;
pp. 119–130.
18. Barroso, L.A.; Hölzle, U. The case for energy-proportional computing. Computer 2007, 40, 33–37. [CrossRef]
19. Buyya, R.; Ranjan, R.; Calheiros, R.N. Modeling and simulation of scalable cloud computing environments and the CloudSim
toolkit: Challenges and opportunities. In Proceedings of the 2009 International Conference on High Performance Computing &
Simulation, Leipzig, Germany, 21–24 June 2009; pp. 1–11.
20. Calheiros, R.N.; Ranjan, R.; Beloglazov, A.; De Rose, C.A.; Buyya, R. CloudSim: A toolkit for modeling and simulation of cloud
computing environments and evaluation of resource provisioning algorithms. Softw. Pract. Exp. 2011, 41, 23–50. [CrossRef]
21. Calheiros, R.N.; Ranjan, R.; De Rose, C.A.F.; Buyya, R. CloudSim: A novel framework for modeling and simulation of cloud
computing infrastructures and services. arXiv 2009, arXiv:0903.2525.
22. Garg, S.K.; Buyya, R. Networkcloudsim: Modelling parallel applications in cloud simulations. In Proceedings of the 2011 Fourth
IEEE International Conference on Utility and Cloud Computing, Washington, DC, USA, 4–9 July 2011; pp. 105–113.
23. Werner, J.; Geronimo, G.; Westphall, C.; Koch, F.; Freitas, R. Simulator improvements to validate the green cloud computing
approach. In Proceedings of the 7th Latin American Network Operations and Management Symposium, Quito, Ecuador, 10–11
October 2011; pp. 1–8.
24. Belalem, G.; Tayeb, F.Z.; Zaoui, W. Approaches to improve the resources management in the simulator CloudSim. In Proceedings
of the International Conference on Information Computing and Applications, Tangshan, China, 15–18 October 2010; Springer:
Berlin, Germany, 2010; pp. 189–196.
25. Li, X.; Jiang, X.; Ye, K.; Huang, P. DartCSim+: Enhanced cloudsim with the power and network models integrated. In Proceedings
of the 2013 IEEE Sixth International Conference on Cloud Computing, Dresden, Germany, 9–12 December 2013; pp. 644–651.
26. Bux, M.; Leser, U. DynamicCloudSim: Simulating heterogeneity in computational clouds. Future Gener. Comput. Syst. 2015, 46,
85–99. [CrossRef]
27. Guérout, T.; Monteil, T.; Da Costa, G.; Calheiros, R.N.; Buyya, R.; Alexandru, M. Energy-aware simulation with DVFS. Simul.
Model. Pract. Theory 2013, 39, 76–91. [CrossRef]
28. Chen, W.; Deelman, E. WorkflowSim: A toolkit for simulating scientific workflows in distributed environments. In Proceedings
of the 2012 IEEE 8th International Conference on E-Science, Chicago, IL, USA, 8–12 October 2012.
29. Wickremasinghe, B. CloudAnalyst: A CloudSim-based tool for modelling and analysis of large scale cloud computing envi-
ronments. In MEDC Project Report; Distrib. Comput. Proj. Csse Dept., University of Melbourne: Victoria, Australia, 2009; pp.
433–659.
30. Wickremasinghe, B.; Calheiros, R.N.; Buyya, R. Cloudanalyst: A cloudsim-based visual modeller for analysing cloud computing
environments and applications. In Proceedings of the 2010 24th IEEE International Conference on Advanced Information
Networking and Applications, Perth, Australia, 20–23 April 2010; pp. 446–452.
31. Kecskemeti, G. DISSECT-CF: A simulator to foster energy-aware scheduling in infrastructure clouds. Simul. Model. Pract. Theory
2015, 58, 188–218. [CrossRef]
32. Tian, W.; Zhao, Y.; Xu, M.; Zhong, Y.; Sun, X. A Toolkit for Modeling and Simulation of Real-Time Virtual Machine Allocation in a
Cloud Data Center. IEEE Trans. Autom. Sci. Eng. 2013, 12, 153–161. [CrossRef]
Energies 2023, 16, 3187 21 of 21

33. Lim, S.H.; Sharma, B.; Nam, G.; Kim, E.K.; Das, C.R. MDCSim: A multi-tier data center simulation, platform. In Proceedings of
the 2009 IEEE International Conference on Cluster Computing and Workshops, New Orleans, LA, USA, 31 August–4 September
2009; pp. 1–9.
34. Kliazovich, D.; Bouvry, P.; Khan, S.U. GreenCloud: A packet-level simulator of energy-aware cloud computing data centers. J.
Supercomput. 2010, 62, 1263–1283. [CrossRef]
35. Hasan, M.S.; Harding, C.; Yu, H.; Griffiths, A. Modeling delay and packet drop in networked control systems using network
simulator NS2. Int. J. Autom. Comput. 2005, 2, 187–194. [CrossRef]
36. Kurowski, K.; Oleksiak, A.; Piatek,
˛ W.; Piontek, T.; Przybyszewski, A.; W˛eglarz, J. DCworms—A tool for simulation of energy
efficiency in distributed computing infrastructures. Simul. Model. Pract. Theory 2013, 39, 135–151. [CrossRef]
37. Sriram, I. SPECI, a simulation tool exploring cloud-scale data centres. In Proceedings of the IEEE International Conference on
Cloud Computing, Bangalore, India, 21–25 September 2009; pp. 381–392.
38. Avetisyan, A.I.; Campbell, R.; Gupta, I.; Heath, M.T.; Ko, S.Y.; Ganger, G.R.; Namgoong, H. Open cirrus: A global cloud computing
testbed. Computer 2010, 43, 35–43. [CrossRef]
39. Campbell, R.H.; Gupta, I.; Heath, M.T.; Ko, S.Y.; Kozuch, M.; Kunze, M.; Soh, Y.C. Open cirrus™ cloud computing testbed:
Federated data centers for open source systems and services research. HotCloud 2009, 9, 1–5.
40. Deng, Q.; Meisner, D.; Bhattacharjee, A.; Wenisch, T.F.; Bianchini, R. Coscale: Coordinating CPU and memory system DVFS in
server systems. In Proceedings of the 45th Annual IEEE International Symposium on Microarchitecture, Vancouver, BC, Canada,
1–5 December 2012; pp. 143–154.
41. Zhang, Y.; Liu, J.; Kandemir, M. Software-directed data access scheduling for reducing disk energy consumption. In Proceedings
of the 2012 IEEE 32nd International Conference on Distributed Computing Systems, Macau, China, 18–21 June 2012; pp. 596–605.

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual
author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to
people or property resulting from any ideas, methods, instructions or products referred to in the content.