Tikam
Tikam
Tikam
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons
Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is
available at
http://creativecommons.org/licenses/by-sa/3.0/
. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must
provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert,
Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift,
Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States
and other countries.
Linux ® is the registered trademark of Linus Torvalds in the United States and other countries.
XFS ® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States
and/or other countries.
MySQL ® is a registered trademark of MySQL AB in the United States, the European Union and
other countries.
Node.js ® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by the
official Joyent Node.js open source or commercial project.
The OpenStack ® Word Mark and OpenStack logo are either registered trademarks/service marks
or trademarks/service marks of the OpenStack Foundation, in the United States and other
countries and are used with the OpenStack Foundation's permission. We are not affiliated with,
endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
Abstract
This documentation collection provides instructions on how to monitor and optimize the
throughput, latency, and power consumption of Red Hat Enterprise Linux 8 in different scenarios.
Table of Contents
Table of Contents
. . . . . . . . . . . . . FEEDBACK
PROVIDING . . . . . . . . . . . . ON
. . . .RED
. . . . .HAT
. . . . .DOCUMENTATION
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6. . . . . . . . . . . . .
.CHAPTER
. . . . . . . . . . 1.. .GETTING
. . . . . . . . . . STARTED
. . . . . . . . . . .WITH
. . . . . .TUNED
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7. . . . . . . . . . . . .
1.1. THE PURPOSE OF TUNED 7
1.2. TUNED PROFILES 7
The default profile 7
Merged profiles 8
The location of profiles 8
The syntax of profile configuration 9
Additional resources 9
1.3. TUNED PROFILES DISTRIBUTED WITH RHEL 9
Real-time profiles 10
1.4. STATIC AND DYNAMIC TUNING IN TUNED 11
1.5. TUNED NO-DAEMON MODE 11
1.6. INSTALLING AND ENABLING TUNED 12
Procedure 12
1.7. LISTING AVAILABLE TUNED PROFILES 12
Procedure 12
Additional resources 13
1.8. SETTING A TUNED PROFILE 13
Prerequisites 13
Procedure 13
Additional resources 14
1.9. DISABLING TUNED 14
Procedure 14
Additional resources 14
1.10. RELATED INFORMATION 14
. . . . . . . . . . . 2.
CHAPTER . . CUSTOMIZING
. . . . . . . . . . . . . . . . TUNED
. . . . . . . . PROFILES
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
..............
2.1. PREREQUISITES 15
2.2. TUNED PROFILES 15
The default profile 15
Merged profiles 15
The location of profiles 16
The syntax of profile configuration 16
Additional resources 16
2.3. INHERITANCE BETWEEN TUNED PROFILES 16
Additional resources 17
2.4. STATIC AND DYNAMIC TUNING IN TUNED 17
2.5. TUNED PLUG-INS 18
Monitoring plug-ins 18
Tuning plug-ins 18
Syntax for plug-ins in Tuned profiles 18
Short plug-in syntax 19
Conflicting plug-in definitions in a profile 19
Functionality not implemented in any plug-in 20
Additional resources 20
2.6. AVAILABLE TUNED PLUG-INS 20
Monitoring plug-ins 20
Tuning plug-ins 20
2.7. VARIABLES AND BUILT-IN FUNCTIONS IN TUNED PROFILES 23
1
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
Variables 24
Functions 24
Additional resources 25
2.8. BUILT-IN FUNCTIONS AVAILABLE IN TUNED PROFILES 25
2.9. CREATING NEW TUNED PROFILES 26
Prerequisites 26
Procedure 26
Additional resources 27
2.10. MODIFYING EXISTING TUNED PROFILES 27
Prerequisites 27
Procedure 27
Additional resources 28
2.11. SETTING THE DISK SCHEDULER USING TUNED 28
Prerequisites 28
Procedure 28
Additional resources 29
2.12. RELATED INFORMATION 29
.CHAPTER
. . . . . . . . . . 3.
. . USING
. . . . . . . .THE
. . . . .WEB
. . . . .CONSOLE
. . . . . . . . . . . FOR
. . . . . SELECTING
. . . . . . . . . . . . .PERFORMANCE
. . . . . . . . . . . . . . . . .PROFILES
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30
..............
Prerequisites 30
Procedure 30
. . . . . . . . . . . 4.
CHAPTER . . .SETTING
. . . . . . . . . THE
. . . . . DISK
. . . . . .SCHEDULER
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .32
..............
4.1. DISK SCHEDULER CHANGES IN RHEL 8 32
4.2. AVAILABLE DISK SCHEDULERS 32
4.3. RECOMMENDED DISK SCHEDULERS FOR DIFFERENT USE CASES 33
4.4. THE DEFAULT DISK SCHEDULER 33
4.5. DETERMINING THE ACTIVE DISK SCHEDULER 33
Procedure 33
4.6. SETTING THE DISK SCHEDULER USING TUNED 33
Prerequisites 34
Procedure 34
Additional resources 35
4.7. SETTING THE DISK SCHEDULER USING UDEV RULES 35
Procedure 35
4.8. TEMPORARILY SETTING A SCHEDULER FOR A SPECIFIC DISK 36
. . . . . . . . . . . 5.
CHAPTER . . TUNING
. . . . . . . . . THE
. . . . . PERFORMANCE
. . . . . . . . . . . . . . . . . OF
. . . .A. .SAMBA
. . . . . . . .SERVER
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .37
..............
Prerequisites 37
5.1. SETTING THE SMB PROTOCOL VERSION 37
Procedure 37
5.2. TUNING SHARES WITH DIRECTORIES THAT CONTAIN A LARGE NUMBER OF FILES 37
Prerequisites 37
Procedure 37
Additional resources 38
5.3. SETTINGS THAT CAN HAVE A NEGATIVE PERFORMANCE IMPACT 38
.CHAPTER
. . . . . . . . . . 6.
. . .OPTIMIZING
. . . . . . . . . . . . .VIRTUAL
. . . . . . . . . .MACHINE
. . . . . . . . . . PERFORMANCE
. . . . . . . . . . . . . . . . . IN
. . .RHEL
. . . . . . 8. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39
..............
6.1. WHAT INFLUENCES VIRTUAL MACHINE PERFORMANCE 39
6.2. OPTIMIZING VIRTUAL MACHINE PERFORMANCE USING TUNED 39
Prerequisites 40
Procedure 40
Additional resources 40
6.3. OPTIMIZING VIRTUAL MACHINE I/O PERFORMANCE 40
2
Table of Contents
. . . . . . . . . . . 7.
CHAPTER . . MANAGING
. . . . . . . . . . . . .POWER
. . . . . . . . .CONSUMPTION
. . . . . . . . . . . . . . . . WITH
. . . . . . POWERTOP
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .54
..............
7.1. THE PURPOSE OF POWERTOP 54
7.2. USING POWERTOP 54
7.2.1. Prerequisites 54
7.2.2. Starting PowerTOP 54
Procedure 54
7.2.3. Calibrating PowerTOP 54
Procedure 54
7.2.4. Setting the measuring interval 55
Procedure 55
7.2.5. Related information 55
7.3. POWERTOP STATISTICS 55
7.3.1. The Overview tab 55
7.3.2. The Idle stats tab 56
7.3.3. The Device stats tab 56
7.3.4. The Tunables tab 56
Additional resources 57
7.4. GENERATING AN HTML OUTPUT 57
Procedure 57
7.5. OPTIMIZING POWER CONSUMPTION 57
7.5.1. Optimizing power consumption using the powertop service 57
Procedure 57
7.5.2. The powertop2tuned utility 57
7.5.3. Optimizing power consumption using the powertop2tuned utility 57
Prerequisites 57
Procedure 58
Additional information 58
3
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
4
Table of Contents
5
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
For simple comments on specific passages, make sure you are viewing the documentation in the
Multi-page HTML format. Highlight the part of text that you want to comment on. Then, click
the Add Feedback pop-up that appears below the highlighted text, and follow the displayed
instructions.
3. Fill in the Description field with your suggestion for improvement. Include a link to the
relevant part(s) of documentation.
6
CHAPTER 1. GETTING STARTED WITH TUNED
Tuned is distributed with a number of predefined profiles for use cases such as:
High throughput
Low latency
Saving power
It is possible to modify the rules defined for each profile and customize how to tune a particular device.
When you switch to another profile or deactivate Tuned, all changes made to the system settings by the
previous profile revert back to their original state.
You can also configure Tuned to react to changes in device usage and adjusts settings to improve
performance of active devices and reduce power consumption of inactive devices.
The profiles provided with Tuned are divided into the following categories:
Power-saving profiles
Performance-boosting profiles
The performance-boosting profiles include profiles that focus on the following aspects:
7
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
Merged profiles
As an experimental feature, it is possible to select more profiles at once. Tuned will try to merge them
during the load.
If there are conflicts, the settings from the last specified profile takes precedence.
The following example optimizes the system to run in a virtual machine for the best performance and
concurrently tunes it for low power consumption, while the low power consumption is the priority:
WARNING
/usr/lib/tuned/
Distribution-specific profiles are stored in the directory. Each profile has its own directory. The profile
consists of the main configuration file called tuned.conf, and optionally other files, for example
helper scripts.
/etc/tuned/
If you need to customize a profile, copy the profile directory into the directory, which is used for
custom profiles. If there are two profiles of the same name, the custom profile located in /etc/tuned/
is used.
8
CHAPTER 1. GETTING STARTED WITH TUNED
Additional resources
NOTE
balanced
The default power-saving profile. It is intended to be a compromise between performance and power
consumption. It uses auto-scaling and auto-tuning whenever possible. The only drawback is the
increased latency. In the current Tuned release, it enables the CPU, disk, audio, and video plugins,
and activates the conservative CPU governor. The radeon_powersave option uses the dpm-
balanced value if it is supported, otherwise it is set to auto.
powersave
A profile for maximum power saving performance. It can throttle the performance in order to
minimize the actual power consumption. In the current Tuned release it enables USB autosuspend,
WiFi power saving, and Aggressive Link Power Management (ALPM) power savings for SATA host
adapters. It also schedules multi-core power savings for systems with a low wakeup rate and
activates the ondemand governor. It enables AC97 audio power saving or, depending on your
system, HDA-Intel power savings with a 10 seconds timeout. If your system contains a supported
Radeon graphics card with enabled KMS, the profile configures it to automatic power saving. On
ASUS Eee PCs, a dynamic Super Hybrid Engine is enabled.
NOTE
In certain cases, the balanced profile is more efficient compared to the powersave
profile.
Consider there is a defined amount of work that needs to be done, for example a video
file that needs to be transcoded. Your machine might consume less energy if the
transcoding is done on the full power, because the task is finished quickly, the
machine starts to idle, and it can automatically step-down to very efficient power save
modes. On the other hand, if you transcode the file with a throttled machine, the
machine consumes less power during the transcoding, but the process takes longer
and the overall consumed energy can be higher.
throughput-performance
A server profile optimized for high throughput. It disables power savings mechanisms and enables
9
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
A server profile optimized for high throughput. It disables power savings mechanisms and enables
sysctl settings that improve the throughput performance of the disk and network IO. CPU governor
is set to performance.
latency-performance
A server profile optimized for low latency. It disables power savings mechanisms and enables sysctl
settings that improve latency. CPU governor is set to performance and the CPU is locked to the low
C states (by PM QoS).
network-latency
A profile for low latency network tuning. It is based on the latency-performance profile. It
additionally disables transparent huge pages and NUMA balancing, and tunes several other network-
related sysctl parameters.
network-throughput
A profile for throughput network tuning. It is based on the throughput-performance profile. It
additionally increases kernel network buffers.
virtual-guest
A profile designed for virtual guests based on the throughput-performance profile that, among
other tasks, decreases virtual memory swappiness and increases disk readahead values. It does not
disable disk barriers.
virtual-host
A profile designed for virtual hosts based on the throughput-performance profile that, among other
tasks, decreases virtual memory swappiness, increases disk readahead values, and enables a more
aggressive value of dirty pages writeback.
oracle
A profile optimized for Oracle databases loads based on throughput-performance profile. It
additionally disables transparent huge pages and modifies other performance-related kernel
parameters. This profile is provided by the tuned-profiles-oracle package.
desktop
A profile optimized for desktops, based on the balanced profile. It additionally enables scheduler
autogroups for better response of interactive applications.
Real-time profiles
Real-time profiles are intended for systems running the real-time kernel. Without a special kernel build,
they do not configure the system to be real-time. On RHEL, the profiles are available from additional
repositories.
realtime
Use on bare-metal real-time systems.
Provided by the tuned-profiles-realtime package, which is available from the RT or NFV repositories.
realtime-virtual-host
Use in a virtualization host configured for real-time.
Provided by the tuned-profiles-nfv-host package, which is available from the NFV repository.
realtime-virtual-guest
Use in a virtualization guest configured for real-time.
Provided by the tuned-profiles-nfv-guest package, which is available from the NFV repository.
10
CHAPTER 1. GETTING STARTED WITH TUNED
Static tuning
Mainly consists of the application of predefined sysctl and sysfs settings and one-shot activation of
several configuration tools such as ethtool.
Dynamic tuning
Watches how various system components are used throughout the uptime of your system. Tuned
adjusts system settings dynamically based on that monitoring information.
For example, the hard drive is used heavily during startup and login, but is barely used later when the
user might mainly work with applications such as web browsers or email clients. Similarly, the CPU
and network devices are used differently at different times. Tuned monitors the activity of these
components and reacts to the changes in their use.
By default, dynamic tuning is disabled. To enable it, edit the /etc/tuned/tuned-main.conf file and
change the dynamic_tuning option to 1. Tuned then periodically analyzes system statistics and uses
them to update your system tuning settings. To configure the time interval in seconds between
these updates, use the update_interval option.
Currently implemented dynamic tuning algorithms try to balance the performance and powersave,
and are therefore disabled in the performance profiles. Dynamic tuning for individual plug-ins can be
enabled or disabled in the Tuned profiles.
On a typical office workstation, the Ethernet network interface is inactive most of the time. Only a
few emails go in and out or some web pages might be loaded.
For those kinds of loads, the network interface does not have to run at full speed all the time, as it
does by default. Tuned has a monitoring and tuning plug-in for network devices that can detect this
low activity and then automatically lower the speed of that interface, typically resulting in a lower
power usage.
If the activity on the interface increases for a longer period of time, for example because a DVD
image is being downloaded or an email with a large attachment is opened, Tuned detects this and
sets the interface speed to maximum to offer the best performance while the activity level is high.
This principle is used for other plug-ins for CPU and disks as well.
By default, no-daemon mode is disabled because a lot of Tuned functionality is missing in this mode,
including:
D-Bus support
Hot-plug support
11
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
To enable no-daemon mode, include the following line in the /etc/tuned/tuned-main.conf file:
daemon = 0
Procedure
$ tuned-adm active
$ tuned-adm verify
Procedure
$ tuned-adm list
Available profiles:
- balanced - General non-specialized tuned profile
- desktop - Optimize for the desktop use-case
- latency-performance - Optimize for deterministic performance at the cost of increased
power consumption
- network-latency - Optimize for deterministic performance at the cost of increased power
12
CHAPTER 1. GETTING STARTED WITH TUNED
$ tuned-adm active
Additional resources
Prerequisites
The tuned service is running. See Section 1.6, “Installing and enabling Tuned” for details.
Procedure
1. Optionally, you can let Tuned recommend the most suitable profile for your system:
# tuned-adm recommend
balanced
2. Activate a profile:
The following example optimizes the system to run in a virtual machine with the best
performance and concurrently tunes it for low power consumption, while the low power
consumption is the priority:
13
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
$ tuned-adm active
$ tuned-adm verify
Additional resources
Procedure
# tuned-adm off
The tunings are applied again after the tuned service restarts.
Additional resources
14
CHAPTER 2. CUSTOMIZING TUNED PROFILES
2.1. PREREQUISITES
Install and enable Tuned as described in Section 1.6, “Installing and enabling Tuned” .
The profiles provided with Tuned are divided into the following categories:
Power-saving profiles
Performance-boosting profiles
The performance-boosting profiles include profiles that focus on the following aspects:
Merged profiles
As an experimental feature, it is possible to select more profiles at once. Tuned will try to merge them
during the load.
If there are conflicts, the settings from the last specified profile takes precedence.
15
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
The following example optimizes the system to run in a virtual machine for the best performance and
concurrently tunes it for low power consumption, while the low power consumption is the priority:
WARNING
/usr/lib/tuned/
Distribution-specific profiles are stored in the directory. Each profile has its own directory. The profile
consists of the main configuration file called tuned.conf, and optionally other files, for example
helper scripts.
/etc/tuned/
If you need to customize a profile, copy the profile directory into the directory, which is used for
custom profiles. If there are two profiles of the same name, the custom profile located in /etc/tuned/
is used.
Additional resources
[main]
include=parent
All settings from the parent profile are loaded in this child profile. In the following sections, the child
16
CHAPTER 2. CUSTOMIZING TUNED PROFILES
All settings from the parent profile are loaded in this child profile. In the following sections, the child
profile can override certain settings inherited from the parent profile or add new settings not present in
the parent profile.
You can create your own child profile in the /etc/tuned/ directory based on a pre-installed profile in
/usr/lib/tuned/ with only some parameters adjusted.
If the parent profile is updated, such as after a Tuned upgrade, the changes are reflected in the child
profile.
The following is an example of a custom profile that extends the balanced profile and sets
Aggressive Link Power Management (ALPM) for all devices to the maximum powersaving.
[main]
include=balanced
[scsi_host]
alpm=min_power
Additional resources
Static tuning
Mainly consists of the application of predefined sysctl and sysfs settings and one-shot activation of
several configuration tools such as ethtool.
Dynamic tuning
Watches how various system components are used throughout the uptime of your system. Tuned
adjusts system settings dynamically based on that monitoring information.
For example, the hard drive is used heavily during startup and login, but is barely used later when the
user might mainly work with applications such as web browsers or email clients. Similarly, the CPU
and network devices are used differently at different times. Tuned monitors the activity of these
components and reacts to the changes in their use.
By default, dynamic tuning is disabled. To enable it, edit the /etc/tuned/tuned-main.conf file and
change the dynamic_tuning option to 1. Tuned then periodically analyzes system statistics and uses
them to update your system tuning settings. To configure the time interval in seconds between
these updates, use the update_interval option.
Currently implemented dynamic tuning algorithms try to balance the performance and powersave,
and are therefore disabled in the performance profiles. Dynamic tuning for individual plug-ins can be
enabled or disabled in the Tuned profiles.
On a typical office workstation, the Ethernet network interface is inactive most of the time. Only a
17
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
On a typical office workstation, the Ethernet network interface is inactive most of the time. Only a
few emails go in and out or some web pages might be loaded.
For those kinds of loads, the network interface does not have to run at full speed all the time, as it
does by default. Tuned has a monitoring and tuning plug-in for network devices that can detect this
low activity and then automatically lower the speed of that interface, typically resulting in a lower
power usage.
If the activity on the interface increases for a longer period of time, for example because a DVD
image is being downloaded or an email with a large attachment is opened, Tuned detects this and
sets the interface speed to maximum to offer the best performance while the activity level is high.
This principle is used for other plug-ins for CPU and disks as well.
monitoring plug-ins
tuning plug-ins
Monitoring plug-ins
Monitoring plug-ins are used to get information from a running system. The output of the monitoring
plug-ins can be used by tuning plug-ins for dynamic tuning.
Monitoring plug-ins are automatically instantiated whenever their metrics are needed by any of the
enabled tuning plug-ins. If two tuning plug-ins require the same data, only one instance of the
monitoring plug-in is created and the data is shared.
Tuning plug-ins
Each tuning plug-in tunes an individual subsystem and takes several parameters that are populated
from the tuned profiles. Each subsystem can have multiple devices, such as multiple CPUs or network
cards, that are handled by individual instances of the tuning plug-ins. Specific settings for individual
devices are also supported.
[NAME]
type=TYPE
devices=DEVICES
NAME
is the name of the plug-in instance as it is used in the logs. It can be an arbitrary string.
TYPE
is the type of the tuning plug-in.
DEVICES
is the list of devices that this plug-in instance handles.
The devices line can contain a list, a wildcard ( *), and negation (!). If there is no devices line, all
18
CHAPTER 2. CUSTOMIZING TUNED PROFILES
The devices line can contain a list, a wildcard ( *), and negation (!). If there is no devices line, all
devices present or later attached on the system of the TYPE are handled by the plug-in instance.
This is same as using the devices=* option.
The following example matches all block devices starting with sd, such as sda or sdb, and does
not disable barriers on them:
[data_disk]
type=disk
devices=sd*
disable_barriers=false
The following example matches all block devices except sda1 and sda2:
[data_disk]
type=disk
devices=!sda1, !sda2
disable_barriers=false
If the plug-in supports more options, they can be also specified in the plug-in section. If the option is not
specified and it was not previously specified in the included plug-in, the default value is used.
[TYPE]
devices=DEVICES
In this case, it is possible to omit the type line. The instance is then referred to with a name, same as the
type. The previous example could be then rewritten into:
[disk]
devices=sdb*
disable_barriers=false
You can also disable the plug-in by specifying the enabled=false option. This has the same effect as if
19
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
You can also disable the plug-in by specifying the enabled=false option. This has the same effect as if
the instance was never defined. Disabling the plug-in is useful if you are redefining the previous
definition from the include option and do not want the plug-in to be active in your custom profile.
You can specify arbitrary shell commands using the script plug-in.
Additional resources
Monitoring plug-ins
Currently, the following monitoring plug-ins are implemented:
disk
Gets disk load (number of IO operations) per device and measurement interval.
net
Gets network load (number of transferred packets) per network card and measurement interval.
load
Gets CPU load per CPU and measurement interval.
Tuning plug-ins
Currently, the following tuning plug-ins are implemented. Only some of these plug-ins implement
dynamic tuning. Options supported by plug-ins are also listed:
cpu
Sets the CPU governor to the value specified by the governor option and dynamically changes the
Power Management Quality of Service (PM QoS) CPU Direct Memory Access (DMA) latency
according to the CPU load.
If the CPU load is lower than the value specified by the load_threshold option, the latency is set to
the value specified by the latency_high option, otherwise it is set to the value specified by
latency_low.
You can also force the latency to a specific value and prevent it from dynamically changing further.
To do so, set the force_latency option to the required latency value.
eeepc_she
Dynamically sets the front-side bus (FSB) speed according to the CPU load.
This feature can be found on some netbooks and is also known as the ASUS Super Hybrid Engine
(SHE).
If the CPU load is lower or equal to the value specified by the load_threshold_powersave option,
the plug-in sets the FSB speed to the value specified by the she_powersave option. If the CPU load
is higher or equal to the value specified by the load_threshold_normal option, it sets the FSB speed
to the value specified by the she_normal option.
Static tuning is not supported and the plug-in is transparently disabled if Tuned does not detect the
20
CHAPTER 2. CUSTOMIZING TUNED PROFILES
Static tuning is not supported and the plug-in is transparently disabled if Tuned does not detect the
hardware support for this feature.
net
Configures the Wake-on-LAN functionality to the values specified by the wake_on_lan option. It
uses the same syntax as the ethtool utility. It also dynamically changes the interface speed according
to the interface utilization.
sysctl
Sets various sysctl settings specified by the plug-in options.
The syntax is name=value, where name is the same as the name provided by the sysctl utility.
Use the sysctl plug-in if you need to change system settings that are not covered by other plug-ins
available in Tuned. If the settings are covered by some specific plug-ins, prefer these plug-ins.
usb
Sets autosuspend timeout of USB devices to the value specified by the autosuspend parameter.
The value 0 means that autosuspend is disabled.
vm
Enables or disables transparent huge pages depending on the Boolean value of the
transparent_hugepages option.
audio
Sets the autosuspend timeout for audio codecs to the value specified by the timeout option.
Currently, the snd_hda_intel and snd_ac97_codec codecs are supported. The value 0 means that
the autosuspend is disabled. You can also enforce the controller reset by setting the Boolean option
reset_controller to true.
disk
Sets the disk elevator to the value specified by the elevator option.
It also sets:
The current disk readahead to a value multiplied by the constant specified by the
readahead_multiply option
In addition, this plug-in dynamically changes the advanced power management and spindown
timeout setting for the drive according to the current drive utilization. The dynamic tuning can be
controlled by the Boolean option dynamic and is enabled by default.
scsi_host
Tunes options for SCSI hosts.
It sets Aggressive Link Power Management (ALPM) to the value specified by the alpm option.
mounts
21
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
Enables or disables barriers for mounts according to the Boolean value of the disable_barriers
option.
script
Executes an external script or binary when the profile is loaded or unloaded. You can choose an
arbitrary executable.
IMPORTANT
The script plug-in is provided mainly for compatibility with earlier releases. Prefer
other Tuned plug-ins if they cover the required functionality.
You need to correctly implement the stop action in your executable and revert all settings that you
changed during the start action. Otherwise, the roll-back step after changing your Tuned profile will
not work.
Bash scripts can import the /usr/lib/tuned/functions Bash library and use the functions defined
there. Use these functions only for functionality that is not natively provided by Tuned. If a function
name starts with an underscore, such as _wifi_set_power_level, consider the function private and do
not use it in your scripts, because it might change in the future.
Specify the path to the executable using the script parameter in the plug-in configuration.
To run a Bash script named script.sh that is located in the profile directory, use:
[script]
script=${i:PROFILE_DIR}/script.sh
sysfs
Sets various sysfs settings specified by the plug-in options.
The syntax is name=value, where name is the sysfs path to use.
Use this plugin in case you need to change some settings that are not covered by other plug-ins.
Prefer specific plug-ins if they cover the required settings.
video
Sets various powersave levels on video cards. Currently, only the Radeon cards are supported.
The powersave level can be specified by using the radeon_powersave option. Supported values are:
default
auto
low
22
CHAPTER 2. CUSTOMIZING TUNED PROFILES
mid
high
dynpm
dpm-battery
dpm-balanced
dpm-perfomance
For details, see www.x.org. Note that this plug-in is experimental and the option might change in
future releases.
bootloader
Adds options to the kernel command line. This plug-in supports only the GRUB 2 boot loader.
Customized non-standard location of the GRUB 2 configuration file can be specified by the
grub2_cfg_file option.
The kernel options are added to the current GRUB configuration and its templates. The system
needs to be rebooted for the kernel options to take effect.
Switching to another profile or manually stopping the tuned service removes the additional options.
If you shut down or reboot the system, the kernel options persist in the grub.cfg file.
For example, to add the quiet kernel option to a Tuned profile, include the following lines in the
tuned.conf file:
[bootloader]
cmdline=quiet
The following is an example of a custom profile that adds the isolcpus=2 option to the kernel
command line:
[bootloader]
cmdline=isolcpus=2
Using Tuned variables reduces the amount of necessary typing in Tuned profiles. You can also:
23
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
Create custom functions in Python and add them to Tuned in the form of plug-ins
Variables
There are no predefined variables in Tuned profiles. You can define your own variables by creating the
[variables] section in a profile and using the following syntax:
[variables]
variable_name=value
${variable_name}
In the following example, the ${isolated_cores} variable expands to 1,2; hence the kernel boots with
the isolcpus=1,2 option:
[variables]
isolated_cores=1,2
[bootloader]
cmdline=isolcpus=${isolated_cores}
The variables can be specified in a separate file. For example, you can add the following lines to
tuned.conf:
[variables]
include=/etc/tuned/my-variables.conf
[bootloader]
cmdline=isolcpus=${isolated_cores}
If you add the isolated_cores=1,2 option to the /etc/tuned/my-variables.conf file, the kernel boots
with the isolcpus=1,2 option.
Functions
To call a function, use the following syntax:
${f:function_name:argument_1:argument_2}
To expand the directory path where the profile and the tuned.conf file are located, use the
PROFILE_DIR function, which requires special syntax:
${i:PROFILE_DIR}
Example 2.9. Isolating CPU cores using variables and built-in functions
In the following example, the ${non_isolated_cores} variable expands to 0,3-5, and the
cpulist_invert built-in function is called with the 0,3-5 argument:
24
CHAPTER 2. CUSTOMIZING TUNED PROFILES
[variables]
non_isolated_cores=0,3-5
[bootloader]
cmdline=isolcpus=${f:cpulist_invert:${non_isolated_cores}}
The cpulist_invert function inverts the list of CPUs. For a 6-CPU machine, the inversion is 1,2, and
the kernel boots with the isolcpus=1,2 command-line option.
Additional resources
PROFILE_DIR
Returns the directory path where the profile and the tuned.conf file are located.
exec
Executes a process and returns its output.
assertion
Compares two arguments. If they do not match, the function logs text from the first argument and
aborts profile loading.
assertion_non_equal
Compares two arguments. If they match, the function logs text from the first argument and aborts
profile loading.
kb2s
Converts kilobytes to disk sectors.
s2kb
Converts disk sectors to kilobytes.
strip
Creates a string from all passed arguments and deletes both leading and trailing white space.
virt_check
Checks whether Tuned is running inside a virtual machine (VM) or on bare metal:
On bare metal, the function returns the second argument, even in case of an error.
cpulist_invert
Inverts a list of CPUs to make its complement. For example, on a system with 4 CPUs, numbered
from 0 to 3, the inversion of the list 0,2,3 is 1.
cpulist2hex
Converts a CPU list to a hexadecimal CPU mask.
cpulist2hex_invert
Converts a CPU list to a hexadecimal CPU mask and inverts it.
25
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
hex2cpulist
Converts a hexadecimal CPU mask to a CPU list.
cpulist_online
Checks whether the CPUs from the list are online. Returns the list containing only online CPUs.
cpulist_present
Checks whether the CPUs from the list are present. Returns the list containing only present CPUs.
cpulist_unpack
Unpacks a CPU list in the form of 1-3,4 to 1,2,3,4.
cpulist_pack
Packs a CPU list in the form of 1,2,3,5 to 1-3,5.
Prerequisites
The tuned service is installed and running. See Section 1.6, “Installing and enabling Tuned” for
details.
Procedure
1. In the /etc/tuned/ directory, create a new directory named the same as the profile that you want
to create:
# mkdir /etc/tuned/my-profile
2. In the new directory, create a file named tuned.conf. Add a [main] section and plug-in
definitions in it, according to your requirements.
For example, see the configuration of the balanced profile:
[main]
summary=General non-specialized tuned profile
[cpu]
governor=conservative
energy_perf_bias=normal
[audio]
timeout=10
[video]
radeon_powersave=dpm-balanced, auto
[scsi_host]
alpm=medium_power
26
CHAPTER 2. CUSTOMIZING TUNED PROFILES
4. Verify that the Tuned profile is active and the system settings are applied:
$ tuned-adm active
$ tuned-adm verify
Additional resources
Prerequisites
The tuned service is installed and running. See Section 1.6, “Installing and enabling Tuned” for
details.
Procedure
1. In the /etc/tuned/ directory, create a new directory named the same as the profile that you want
to create:
# mkdir /etc/tuned/modified-profile
2. In the new directory, create a file named tuned.conf, and set the [main] section as follows:
[main]
include=parent-profile
Replace parent-profile with the name of the profile you are modifying.
To use the settings from the throughput-performance profile and change the value of
vm.swappiness to 5, instead of the default 10, use:
[main]
include=throughput-performance
[sysctl]
vm.swappiness=5
27
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
5. Verify that the Tuned profile is active and the system settings are applied:
$ tuned-adm active
$ tuned-adm verify
Additional resources
device with the name of the block device, for example sdf
selected-scheduler with the disk scheduler that you want to set for the device, for example bfq
Prerequisites
Procedure
1. Optional: Select an existing Tuned profile on which your profile will be based. For a list of
available profiles, see Section 1.3, “Tuned profiles distributed with RHEL” .
To see which profile is currently active, use:
$ tuned-adm active
# mkdir /etc/tuned/my-profile
3. Find the World Wide Name (WWN) identifier of the selected block device:
ID_WWN=0x5002538d00000000
4. Create the /etc/tuned/my-profile/tuned.conf configuration file. In the file, set the following
options:
28
CHAPTER 2. CUSTOMIZING TUNED PROFILES
[main]
include=existing-profile
Set the selected disk scheduler for the device that matches the WWN identifier:
[disk]
devices_udev_regex=ID_WWN=0x5002538d00000000
elevator=selected-scheduler
To match multiple devices in the devices_udev_regex option, separate the identifiers with
commas:
devices_udev_regex=ID_WWN=0x5002538d00000000,
ID_WWN=0x1234567800000000
$ tuned-adm active
$ tuned-adm verify
Additional resources
For more information on creating a Tuned profile, see Chapter 2, Customizing Tuned profiles .
29
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
Latency performance
Network performance
Virtual machines
The following procedure describes setting up performance profiles in the web console.
For details about the tuned service, see Monitoring and managing system status and performance .
Prerequisites
The web console must be installed and accessible.
For details, see Installing the web console .
Procedure
1. Log in to the RHEL 8 web console.
For details, see Logging in to the web console .
2. Click System.
4. In the Change Performance Profile dialog box, change the profile if necessary.
5. Click Change.
30
CHAPTER 3. USING THE WEB CONSOLE FOR SELECTING PERFORMANCE PROFILES
31
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
Set the scheduler using Tuned, as described in Section 4.6, “Setting the disk scheduler using
Tuned”
Set the scheduler using udev, as described in Section 4.7, “Setting the disk scheduler using
udev rules”
The traditional, single-queue schedulers, which were available in RHEL 7 and earlier versions, have been
removed.
Disk schedulers
none
Implements a first-in first-out (FIFO) scheduling algorithm. It merges requests at the generic block
layer through a simple last-hit cache.
mq-deadline
Attempts to provide a guaranteed latency for requests from the point at which requests reach the
scheduler.
The mq-deadline scheduler sorts queued I/O requests into a read or write batch and then schedules
them for execution in increasing logical block addressing (LBA) order. By default, read batches take
precedence over write batches, because applications are more likely to block on read I/O operations.
After mq-deadline processes a batch, it checks how long write operations have been starved of
processor time and schedules the next read or write batch as appropriate.
This scheduler is suitable for most use cases, but particularly those in which read operations occur
more often than write operations.
bfq
Targets desktop systems and interactive tasks.
The bfq scheduler ensures that a single application is never using all of the bandwidth. In effect, the
storage device is always as responsive as if it was idle. The system does not become unresponsive
when copying large files. In its default configuration, bfq focuses on delivering the lowest latency
rather than achieving the maximum throughput.
bfq is based on cfq code. It does not grant the disk to each process for a fixed time slice but assigns a
budget measured in number of sectors to the process.
32
CHAPTER 4. SETTING THE DISK SCHEDULER
kyber
Is intended for fast devices. The scheduler tunes itself to achieve a latency goal. You can configure
the target latencies for read and synchronous write requests.
High-performance SSD or a CPU-bound system with Use none, especially when running enterprise
fast storage applications. Alternatively, use kyber.
The kernel selects a default disk scheduler based on the type of device. The automatically selected
scheduler is typically the optimal setting. If you require a different scheduler, Red Hat recommends to
use udev rules or the Tuned application to configure it. Match the selected devices and switch the
scheduler only for those devices.
Procedure
# cat /sys/block/device/queue/scheduler
In the file name, replace device with the block device name, for example sdc.
33
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
device with the name of the block device, for example sdf
selected-scheduler with the disk scheduler that you want to set for the device, for example bfq
Prerequisites
Procedure
1. Optional: Select an existing Tuned profile on which your profile will be based. For a list of
available profiles, see Section 1.3, “Tuned profiles distributed with RHEL” .
To see which profile is currently active, use:
$ tuned-adm active
# mkdir /etc/tuned/my-profile
3. Find the World Wide Name (WWN) identifier of the selected block device:
ID_WWN=0x5002538d00000000
4. Create the /etc/tuned/my-profile/tuned.conf configuration file. In the file, set the following
options:
[main]
include=existing-profile
Set the selected disk scheduler for the device that matches the WWN identifier:
[disk]
devices_udev_regex=ID_WWN=0x5002538d00000000
elevator=selected-scheduler
To match multiple devices in the devices_udev_regex option, separate the identifiers with
commas:
devices_udev_regex=ID_WWN=0x5002538d00000000,
ID_WWN=0x1234567800000000
34
CHAPTER 4. SETTING THE DISK SCHEDULER
$ tuned-adm active
$ tuned-adm verify
Additional resources
For more information on creating a Tuned profile, see Chapter 2, Customizing Tuned profiles .
device with the name of the block device, for example sdf
selected-scheduler with the disk scheduler that you want to set for the device, for example bfq
Procedure
ATTRS{wwid}=="device WWID"
2. Configure the udev rule. Create the /etc/udev/rules.d/99-scheduler.rules file with the
following content:
Replace device WWID with the WWID value that you found in the previous steps.
35
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
# cat /sys/block/device/queue/scheduler
Procedure
In the file name, replace device with the block device name, for example sdc.
Verification steps
# cat /sys/block/device/queue/scheduler
36
CHAPTER 5. TUNING THE PERFORMANCE OF A SAMBA SERVER
Parts of this section were adopted from the Performance Tuning documentation published in the Samba
Wiki. License: CC BY 4.0. Authors and contributors: See the history tab on the Wiki page.
Prerequisites
Samba is set up as a file or print server
See Using Samba as a server .
NOTE
To always have the latest stable SMB protocol version enabled, do not set the server
max protocol parameter. If you set the parameter manually, you will need to modify the
setting with each new version of the SMB protocol, to have the latest protocol version
enabled.
The following procedure explains how to use the default value in the server max protocol parameter.
Procedure
1. Remove the server max protocol parameter from the [global] section in the
/etc/samba/smb.conf file.
Prerequisites
Procedure
37
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
NOTE
Using the settings in this procedure, files with names other than in lowercase will
no longer be displayed.
For details about the parameters, see their descriptions in the smb.conf(5) man page.
# testparm
After you applied these settings, the names of all newly created files on this share use lowercase.
Because of these settings, Samba no longer needs to scan the directory for uppercase and lowercase,
which improves the performance.
Additional resources
To use the optimized settings from the Kernel, remove the socket options parameter from the [global]
section in the /etc/samba/smb.conf.
38
CHAPTER 6. OPTIMIZING VIRTUAL MACHINE PERFORMANCE IN RHEL 8
Virtual CPUs (vCPUs) are implemented as threads on the host, handled by the Linux scheduler.
VMs do not automatically inherit optimization features such as NUMA and huge pages from the
host kernel.
Disk and network I/O settings of the host might have a significant performance impact on the
VM.
Depending on the host devices and their models, there might be significant overhead due to
emulation of particular hardware.
Nevertheless, a variety of features are available that you can use to reduce the negative performance
effects of virtualization. Notably:
The tuned service can automatically optimize the resource distribution and performance of
your VMs.
Block I/O tuning can improve the performances of the VM’s block devices, such as disks.
NOTE
Tuning VM performance can have adverse effects on other virtualization functions. For
example, it can make migrating the modified VM more difficult.
39
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
Red Hat recommends using the following profiles when using virtualization in RHEL 8:
For RHEL 8 virtual machines, use the virtual-guest profile. It is based on the generally
applicable throughput-performance profile, but also decreases the swappiness of virtual
memory.
For RHEL 8 virtualization hosts, use the virtual-host profile. This enables more aggressive
writeback of dirty memory pages, which benefits the host performance.
Prerequisites
Procedure
To enable a specific tuned profile:
# tuned-adm list
Available profiles:
- balanced - General non-specialized tuned profile
- desktop - Optimize for the desktop use-case
[...]
- virtual-guest - Optimize for running inside a virtual guest
- virtual-host - Optimize for running KVM guests
Current active profile: balanced
Additional resources
For more information about tuned and tuned profiles, see Monitoring and managing system
status and performance.
40
CHAPTER 6. OPTIMIZING VIRTUAL MACHINE PERFORMANCE IN RHEL 8
Increasing the I/O weight of a device increases its priority for I/O bandwidth, and therefore provides it
with more host resources. Similarly, reducing a device’s weight makes it consume less host resources.
NOTE
Each device’s weight value must be within the 100 to 1000 range. Alternatively, the
value can be `0, which removes that device from per-device listings.
Procedure
To display and set a VM’s block I/O parameters:
<domain>
...
<blkiotune>
<weight>800</weight>
<device>
<path>/dev/sda</path>
<weight>1000</weight>
</device>
<device>
<path>/dev/sdb</path>
<weight>500</weight>
</device>
</blkiotune>
...
</domain>
For example, the following changes the weight of the /dev/sda device in the liftrul VM to 500.
To enable disk I/O throttling, set a limit on disk I/O requests sent from each block device attached to
VMs to the host machine.
Procedure
41
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
1. Use the virsh domblklist command for a list of disk device names on a specified VM.
2. Set I/O limits for a block device attached to a VM using the virsh blkdeviotune command:
For example, to throttle the sdb device on the rollin-coal VM to 1000 I/O operations per
second and 50 MB per second throughput:
Additional information
Disk I/O throttling can be useful in various situations, for example when VMs belonging to
different customers are running on the same host, or when quality of service guarantees are
given for different VMs. Disk I/O throttling can also be used to simulate slower disks.
I/O throttling can be applied independently to each block device attached to a VM and
supports limits on throughput and I/O operations.
Procedure
To enable multi-queue virtio-scsi support for a specific VM, add the following to the VM’s XML
configuration, where N is the total number of vCPU queues:
1. Ensure that the vCPU model is aligned with the CPU model of the host. For example, to set the
testguest1 VM to use the CPU model of the host:
2. If your host machine uses Non-Uniform Memory Access (NUMA), you can also configure NUMA
42
CHAPTER 6. OPTIMIZING VIRTUAL MACHINE PERFORMANCE IN RHEL 8
for its VMs. This maps the host’s CPU and memory processes to CPU and memory processes
on the VM as closely as possible. In effect, NUMA tuning provides the vCPU with a more
streamlined access to the system memory allocated to the VM, which can improve the vCPU
processing effectiveness.
For details, see Section 6.4.1, “Configuring NUMA in a virtual machine” and Section 6.4.2,
“Sample vCPU performance tuning scenario”.
Prerequisites
The host must be a NUMA-compatible machine. To detect whether this is the case, use the
virsh nodeinfo command and see the NUMA cell(s) line:
# virsh nodeinfo
CPU model: x86_64
CPU(s): 48
CPU frequency: 1200 MHz
CPU socket(s): 1
Core(s) per socket: 12
Thread(s) per core: 2
NUMA cell(s): 2
Memory size: 67012964 KiB
Procedure
For ease of use, you can set up a VM’s NUMA configuration using automated utilities and services.
However, manual NUMA setup is more likely to yield a significant performance improvement.
Automatic methods
Set the VM’s NUMA policy to Preferred. For example, to do so for the testguest5 VM:
Use the numad command to automatically align the VM CPU with memory resources.
# numad
Manual methods
1. Pin specific vCPU threads to a specific host CPU or range of CPUs. This is possible also on non-
NUMA hosts and VMs, and is recommended as a safe method of vCPU performance
improvement.
For example, the following commands pin vCPU threads 0 to 5 of the testguest6 VM to host
CPUs 1, 3, 5, 7, 9, and 11, respectively:
43
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
2. After pinning vCPU threads, you can also pin QEMU process threads associated with a specified
VM to a specific host CPU or range of CPUs. For example, the following commands pin the
QEMU process thread of testguest6 to CPUs 13 and 15, and verify this was successful:
3. Finally, you can also specify which host NUMA nodes will be assigned specifically to a certain
VM. This can improve the host memory usage by the VM’s vCPU. For example, the following
commands set testguest6 to use host NUMA nodes 3 to 5, and verify this was successful:
Additional resources
Note that for best performance results, it is recommended to use all of the manual tuning
methods listed above. For an example of such a configuration, see Section 6.4.2, “Sample vCPU
performance tuning scenario”.
To see the current NUMA configuration of your system, you can use the numastat utility. For
details on using numastat, see Section 6.6, “Virtual machine performance monitoring tools” .
Starting scenario
2 NUMA nodes
44
CHAPTER 6. OPTIMIZING VIRTUAL MACHINE PERFORMANCE IN RHEL 8
The output of virsh nodeinfo of such a machine would look similar to:
# virsh nodeinfo
CPU model: x86_64
CPU(s): 12
CPU frequency: 3661 MHz
CPU socket(s): 2
Core(s) per socket: 3
Thread(s) per core: 2
NUMA cell(s): 2
Memory size: 31248692 KiB
You intend to modify an existing VM to have 8 vCPUs, which means that it will not fit in a single
NUMA node.
Therefore, you should distribute 4 vCPUs on each NUMA node and make the vCPU topology
resemble the host topology as closely as possible. This means that vCPUs that run as sibling
threads of a given physical CPU should be pinned to host threads on the same core. For details,
see the Solution below:
Solution
# virsh capabilities
The output should include a section that looks similar to the following:
<topology>
<cells num="2">
<cell id="0">
<memory unit="KiB">15624346</memory>
<pages unit="KiB" size="4">3906086</pages>
<pages unit="KiB" size="2048">0</pages>
<pages unit="KiB" size="1048576">0</pages>
<distances>
<sibling id="0" value="10" />
<sibling id="1" value="21" />
</distances>
<cpus num="6">
<cpu id="0" socket_id="0" core_id="0" siblings="0,3" />
<cpu id="1" socket_id="0" core_id="1" siblings="1,4" />
<cpu id="2" socket_id="0" core_id="2" siblings="2,5" />
<cpu id="3" socket_id="0" core_id="0" siblings="0,3" />
<cpu id="4" socket_id="0" core_id="1" siblings="1,4" />
<cpu id="5" socket_id="0" core_id="2" siblings="2,5" />
</cpus>
</cell>
<cell id="1">
<memory unit="KiB">15624346</memory>
<pages unit="KiB" size="4">3906086</pages>
<pages unit="KiB" size="2048">0</pages>
45
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
2. [Optional] Test the performance of the VM using the applicable tools and utilities.
default_hugepagesz=1G hugepagesz=1G
[Unit]
Description=HugeTLB Gigantic Pages Reservation
DefaultDependencies=no
Before=dev-hugepages.mount
ConditionPathExists=/sys/devices/system/node
ConditionKernelCommandLine=hugepagesz=1G
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/etc/systemd/hugetlb-reserve-pages.sh
[Install]
WantedBy=sysinit.target
#!/bin/sh
nodes_path=/sys/devices/system/node/
if [ ! -d $nodes_path ]; then
echo "ERROR: $nodes_path does not exist"
exit 1
fi
reserve_pages()
{
46
CHAPTER 6. OPTIMIZING VIRTUAL MACHINE PERFORMANCE IN RHEL 8
reserve_pages 4 node1
reserve_pages 4 node2
This reserves four 1GiB huge pages from node1 and four 1GiB huge pages from node2.
# chmod +x /etc/systemd/hugetlb-reserve-pages.sh
4. Use the virsh edit command to edit the XML configuration of the VM you wish to optimize, in
this example super-VM:
Set the VM to use 8 static vCPUs. Use the <vcpu/> element to do this.
Pin each of the vCPU threads to the corresponding host CPU threads that it mirrors in
topology. To do so, use the <vcpupin/> elements in the <cputune> section.
Note that, as shown by virsh capabilities above, host CPU threads are not ordered
sequentially in their respective cores. In addition, the vCPU threads should be pinned to the
highest available set of host cores on the same NUMA node. For a table illustration, see the
Additional Resources section below.
Configure the VM’s NUMA nodes to use memory from the corresponding NUMA nodes on
the host. To do so, use the <memnode/> elements in the <numatune/> section.
Ensure the CPU mode is set to host-passthrough, and that the CPU uses cache in
passthrough mode.
The resulting XML configuration should include a section similar to the following:
[...]
<memoryBacking>
<hugepages>
<page size='1' unit='GiB'/>
</hugepages>
</memoryBacking>
<vcpu placement='static'>8</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='1'/>
<vcpupin vcpu='1' cpuset='4'/>
<vcpupin vcpu='2' cpuset='2'/>
<vcpupin vcpu='3' cpuset='5'/>
<vcpupin vcpu='4' cpuset='7'/>
<vcpupin vcpu='5' cpuset='10'/>
<vcpupin vcpu='6' cpuset='8'/>
47
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
6. [Optional] Test the performance of the VM using the applicable tools and utilities to evaluate
the impact of the VM’s optimization.
Additional resources
The following tables illustrate the connections between the vCPUs and the host CPUs they
should be pinned to:
CPU threads 0 3 1 4 2 5 6 9 7 10 8 11
Cores 0 1 2 3 4 5
Sockets 0 1
NUMA nodes 0 1
vCPU threads 0 1 2 3 4 5 6 7
Cores 0 1 2 3
Sockets 0 1
48
CHAPTER 6. OPTIMIZING VIRTUAL MACHINE PERFORMANCE IN RHEL 8
NUMA nodes 0 1
vCPU threads 0 1 2 3 4 5 6 7
Host CPU 0 3 1 4 2 5 6 9 7 10 8 11
threads
Cores 0 1 2 3 4 5
Sockets 0 1
NUMA nodes 0 1
In this scenario, there are 2 NUMA nodes and 8 vCPUs. Therefore, 4 vCPU threads should be
pinned to each node.
In addition, Red Hat recommends leaving at least a single CPU thread available on each node
for host system operations.
Because in this example, each NUMA node houses 3 cores, each with 2 host CPU threads, the
set for node 0 translates as follows:
Procedure
Use any of the following methods and observe if it has a beneficial effect on your VM network
performance:
If the output of this command is blank, enable the vhost_net kernel module:
49
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
# modprobe vhost_net
<interface type='network'>
<source network='default'/>
<model type='virtio'/>
<driver name='vhost' queues='N'/>
</interface>
Also, enabling zero-copy transmit can cause head-of-line blocking of packets, which may create a
potential security risk.
# modprobe -r vhost_net
2. Re-enable the vhost-net module with the zero-copy parameter turned on:
# cat /sys/module/vhost_net/parameters/experimental_zcopytx
1
Additional resources
For additional information on virtual network connection types and tips for usage, see
Understanding virtual networking .
50
CHAPTER 6. OPTIMIZING VIRTUAL MACHINE PERFORMANCE IN RHEL 8
On your RHEL 8 host, as root, use the top utility or the system monitor application, and look for
qemu and virt in the output. This shows how much host system resources your VMs are
consuming.
If the monitoring tool displays that any of the qemu or virt processes consume a large
portion of the host CPU or memory capacity, use the perf utility to investigate. For details,
see below.
On the guest operating system, use performance utilities and applications available on the
system to evaluate which processes consume the most system resources.
perf kvm
You can use the perf utility to collect and analyze virtualization-specific statistics about the
performance of your RHEL 8 host. To do so:
2. Use the perf kvm stat command to display perf statistics for your virtualization host:
For real-time monitoring of your hypervisor, use the perf kvm stat live command.
To log the perf data of your hypervisor over a period of time, activate the logging using the
perf kvm stat record command. After the command is canceled or interrupted, the data is
saved in the perf.data.guest file, which can be analyzed using the perf kvm stat report
command.
3. Analyze the perf output for types of VM-EXIT events and their distribution. For example, the
PAUSE_INSTRUCTION events should be infrequent, but in the following output, the high
occurrence of this event suggests that the host CPUs are not handling the running vCPUs well.
In such a scenario, consider powering down some of your active VMs, removing vCPUs from
these VMs, or tuning the performance of the vCPUs.
VM-EXIT Samples Samples% Time% Min Time Max Time Avg time
51
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
Other event types that can signal problems in the output of perf kvm stat include:
For more information on using perf to monitor virtualization performance, see the perf-kvm man page.
numastat
To see the current NUMA configuration of your system, you can use the numastat utility, which is
provided by installing the numactl package.
The following shows a host with 4 running VMs, each obtaining memory from multiple NUMA nodes. This
is not optimal for vCPU performance, and warrants adjusting:
# numastat -c qemu-kvm
In contrast, the following shows memory being provided to each VM by a single node, which is
significantly more efficient.
# numastat -c qemu-kvm
52
CHAPTER 6. OPTIMIZING VIRTUAL MACHINE PERFORMANCE IN RHEL 8
53
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
The PowerTOP tool can provide an estimate of the total power usage of the system and also individual
power usage for each process, device, kernel worker, timer, and interrupt handler. The tool can also
identify specific components of kernel and user-space applications that frequently wake up the CPU.
7.2.1. Prerequisites
To be able to use PowerTOP, make sure that the powertop package has been installed on your
system:
# powertop
IMPORTANT
Laptops should run on battery power when running the powertop command.
1. On a laptop, you can calibrate the power estimation engine by running the following command:
# powertop --calibrate
2. Let the calibration finish without interacting with the machine during the process.
Calibration takes time because the process performs various tests, cycles through brightness
levels and switches devices on and off.
3. When the calibration process is completed, PowerTOP starts as normal. Let it run for
approximately an hour to collect data.
When enough data is collected, power estimation figures will be displayed in the first column of
54
CHAPTER 7. MANAGING POWER CONSUMPTION WITH POWERTOP
When enough data is collected, power estimation figures will be displayed in the first column of
the output table.
NOTE
If you want to change this measuring frequency, use the following procedure:
Procedure
Overview
Idle stats
Frequency stats
Device stats
Tunables
You can use the Tab and Shift+Tab keys to cycle through these tabs.
The adjacent columns within the Overview tab provide the following pieces of information:
Usage
Power estimation of how the resource is being used.
Events/s
Wakeups per second. The number of wakeups per second indicates how efficiently the services or
55
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
Wakeups per second. The number of wakeups per second indicates how efficiently the services or
the devices and drivers of the kernel are performing. Less wakeups means that less power is
consumed. Components are ordered by how much further their power usage can be optimized.
Category
Classification of the component; such as process, device, or timer.
Description
Description of the component.
If properly calibrated, a power consumption estimation for every listed item in the first column is shown
as well.
Apart from this, the Overview tab includes the line with summary statistics such as:
Summary of total wakeups per second, GPU operations per second, and virtual file system
operations per second
Use the up and down keys to move through suggestions, and the enter key to toggle the suggestion on
or off.
56
CHAPTER 7. MANAGING POWER CONSUMPTION WITH POWERTOP
Additional resources
For more details on PowerTOP, see PowerTOP’s home page.
Procedure
# powertop --html=htmlfile.html
Replace the htmlfile.html parameter with the required name for the output file.
Procedure
By default, powertop2tuned creates profiles in the /etc/tuned/ directory, and bases the custom profile
on the currently selected Tuned profile. For safety reasons, all PowerTOP tunings are initially disabled
in the new profile.
Use the --enable or -e option to generate a new profile that enables most of the tunings
suggested by PowerTOP.
Certain potentially problematic tunings, such as the USB autosuspend, are disabled by default
and need to be uncommented manually.
57
Red Hat Enterprise Linux 8 Monitoring and managing system status and performance
Procedure
# powertop2tuned new_profile_name
Additional information
$ powertop2tuned --help
The powertop2tuned utility represents integration of PowerTOP into Tuned, which enables to
benefit of advantages of both tools.
58