The Hidden Threat: Analysis of Linux Rootkit Techniques and Limitations of Current Detection Tools

Already in 2004, insights into Linux-based rootkits were provided by [1]. In his work, the author describes techniques implemented by user and kernel space rootkits as well as detection methods and countermeasures. While this work serves as a first step in this area, it is limited by the small size of rootkits analyzed. To further aid in their detection, Levine et al. propose a categorization for rootkits [4]. Though their work explains the basics of kernel level rootkits in detail, it focuses primarily on those targeting the system table in Linux. Furthermore, their work does not consider recent developments in this area, evident in their proposed kern\(\_\)check utility, which only supports Linux kernels 2.2.x and 2.4.x. Additionally, alternative methods for rootkit detection have been explored, such as leveraging hardware-assisted infrastructure to identify anomalies in intraprocess execution [9] or employing a lightweight hypervisor for filtering rootkit activities [5]. However, these proactive strategies are not typically suited for DFIR applications.

In 2011, Joy et al. provided an overview of various detection methods for rootkits in their study [3]. While offering valuable insights into rootkits and potential detection approaches, including host-based and virtualization-based methods, their discussion lacks depth in evaluating the effectiveness of these approaches. Another study focuses on the analysis of a variety of tools for rootkit detection specifically within forensic investigations [8]. The authors ultimately determine that the assessed rootkit detection tools demonstrate high effectiveness in identifying rootkits on live systems. While this research marks an important initial step toward establishing forensic readiness for rootkit detection, its age and the limited scope of only five rootkits evaluated necessitate an updated exploration.

Other research specifically focused on rootkit detection in the realm of digital forensics remains limited. One notable study addressed the challenge of rootkits concealing themselves within memory forensics [7]. The authors developed an algorithm to locate paging structures in x86 system memory images. These structures, which are used by processes to define their address space, can be leveraged to extract a complete process list from memory, potentially revealing hidden processes, including rootkits. Another significant contribution was made in the field of live system analysis [6], where a detection system was proposed to identify discrepancies in userspace utility outputs, aiding in the discovery of hidden process IDs or files. Although this approach shows promise, its efficacy in userspace is somewhat constrained due to the limited scope of rootkits examined and does not incorporate findings from other analyses, such as storage analysis. In the realm of memory forensics, Case et al. provide an approach for detecting kprobe, ftrace, and Extended Berkley Packets Filters (eBPF)-based hooks using their own custom Volatility plugins [2]. While the article provides valuable insights, it does not include a thorough assessment of the plugins discussed in relation to actual rootkits. Additionally, we were unable to obtain the plugins for our own analysis during the preparation of this article since they were not publicly released.

Our findings suggest that prior research has predominantly concentrated on rootkit prevention, rather than detection, as required in DFIR contexts. Furthermore, a considerable amount of the literature in this domain is now outdated. Consequently, there’s a notable absence of established guidelines for analysts to detect these hidden threats. To address this, the following sections will detail common rootkit techniques and features identified in our analysis.

Like most modern operating systems, Linux implements multiple so-called “protection domains,” which define and control the access programs have to specific system resources. This hierarchical structure is often visualized as concentric rings, with the highest privileged domain in the center. The outermost ring (3) is commonly referred to as the user space while the three inner rings (0–2) form the kernel space. Communication between these domains is carried out via well-defined interfaces. The inner domains and the interfaces to them will usually dictate what information is passed on to the outer domains. Rootkits can abuse this hierarchical flow of information to alter the system state presented to the user.

Inside this structure, rootkits may be classified by three characteristics: Which domain they choose to reside in, how they assert their control in the chosen domain to change information flowing through it, and what specific changes they perform on the information. The following sections will deal with the first two characteristics, explaining the two primary domains rootkits may reside in on a Linux system and how they can insert themselves into control sequences inside these domains to manipulate information.

The user space is the least privileged domain. Here is where user applications and libraries reside.

The kernel space sits “below” the user space and forms a bridge to the underlying hardware. The Linux kernel is responsible for process scheduling, memory management, networking, virtual file systems (VFS), and more.

Most rootkits will have to implement some kind of persistence mechanism to survive potential system reboots. Not all rootkits offer such mechanisms out of the box and leave this task to the attacker. The following are a few examples utilized by the analyzed rootkits.

Application-level rootkits are already persistent by design since they directly replace application binaries. Dynamic linker rootkits using LD\(\_\)PRELOAD mostly use the file /etc/ld.so.preload, which has the same effect as setting the environment variable, just system-wide and persistent. Kernel space rootkits will generally have to find a way to have their LKM loaded into the kernel at system startup. One option can be adding an entry to /etc/modules or /etc/modules-load.d/, which is the intended method of loading LKMs during startup. Rootkits may also use many other methods of persistence, which are not exclusive to rootkits or malware in general. Those could include scripts in cron, systemd, SysVinit, Upstart, or udev.

Table 1 gives an overview of the analyzed rootkits and illustrates the primary features they implement. These feature categories will be explained in the following sections. The table is divided into user space (top) and kernel space (bottom) rootkits. The rootkits were chosen either because of their popularity or interesting/unique features they implement. They cover a wide spectrum of functionality and techniques to properly challenge detection approaches and provide a comprehensive evaluation. We mostly used working versions of the rootkits. Due to the wide range of rootkits, implemented features, and differing release dates, it was not possible to run all samples under the same kernel version. While new features and security measures were added to the kernel between versions, we do not believe these changes alter the detection results of the tools evaluated in this article. We did, however, not use any version which was no longer officially supported as of the writing of this article (\({\lt}\)4.14). In some exceptions, a working version was not available or the required kernel version too old. In these cases, we based our evaluation on an in-depth analysis of the source code of the rootkit or a sufficiently technical and detailed write-up, as in the case of the Russian Dovorub rootkit. These rootkits are marked with alphabet (a) in the following table. Rootkits, where no working version, source code, or detailed analysis report was available, were excluded from our study due to lack of confirmability.

The landscape presented shows how new rootkits are developed and released every year, proving the ongoing need for capable detection solutions.

The following subsections present the core functionalities identified in our study.

Arguably the most well known of the presented tools, rkhunter was first released in 2003 and has received updates until 2018. It offers a wide variety of different submodules, which may be used independently or in unison. Some of these modules require the user to have generated an integrity database before the infection. rkhunter includes modules to check for the integrity of critical system applications, known rootkit paths and kernel symbols, known persisted LKMs, network sniffers, suspicious processes, preloaded libraries, changes in user/group privileges, and many more.

chkrootkit was initially released in 1997 and is frequently updated until today. It consists of three primary utilities: The first (chkrootkit) searches for artifacts of known rootkits (e.g., file paths or strings). The second (chkproc) uses a brute force method, in which it iterates through all possible process IDs and attempts to access their entries in the procfs. If an entry is accessible, but its corresponding process does not appear in the outputs of ps, it might be hidden by a rootkit. The third utility (chkdir) iterates through the file system and searches for directories with an anomalous number of hard links. Many rootkits hide directories but do not modify the number of hard links pointing to the parent directory. So, a hidden directory might be present if the number of hard links does not match the number of visible subdirectories. Additionally, chkrootkit offers multiple tools for log file verification and a script to check for promiscuous network adapters.

Unhide is a forensic tool to unveil hidden processes and network connections. Its most recent version is from January 2021. Unhide’s hidden process scan supports multiple modes, which all function by iterating through the number of available process IDs and interacting with these IDs in different ways. This may include accessing their directories in the procfs or attempting to use them in different functions and system calls. The list of responsive IDs is then matched against the outputs of ps to reveal potentially hidden processes. The hidden network connection scanner follows a similar approach: Unhide attempts to bind itself to all possible port numbers. If a binding is unsuccessful, meaning the port is already in use, it verifies if the corresponding connection is shown in tools like netstat or ss. Should this not be the case, a rootkit might hide the connection.

Tripwire allows an administrator to create a highly customizable snapshot of a Linux system, which can later be used to detect malicious changes to this system. Since the open source development of Tripwire stopped in 2018, the Advanced Intrusion Detection Environment (AIDE) has become its successor (current version from June 2023). Like the integrity checks performed by rkhunter, AIDE requires the victim to have proactively generated an image before infection. This image must have also been protected against manipulation by separating it from the system or encryption. Files must be visible for AIDE to interact with them.

OSSEC and its fork Wazuh are intrusion detection systems that guard endpoints against attacks. They both offer the same rootkit scanner called Rootcheck and are identical in their anti-rootkit capabilities. The most recent version of OSSEC was released in January 2022. The full rootkit check consists of seven stages, including checks for known paths, strings, and devices, a slightly simplified version of Unhide’s process and port checks, and inspection of certain file permissions and network interfaces.

ClamAV is a general-purpose malware scanner (primarily for Linux systems) initially published in 2001. It is currently maintained and regularly updated by Cisco. Malware is detected by scanning directories and comparing found files with an extensive signature database. This comes with the apparent limitation of only being able to detect already known malware and only if their files can be accessed.

LKRG is a mostly proactive utility designed to detect exploits and integrity violations inside the Linux kernel. Its newest version was published in Juli 2023. LKRG is loaded into the kernel as an LKM. Its primary function is to validate the integrity of various kernel structures against a previously generated hash database. In addition, it performs various checks for potential kernel exploits. This includes the verification of credential structures to detect processes with illegitimately elevated privileges, as well as a comparison of the kernel modules exported via /proc/modules and the list of kobjects in /sys/module. Discrepancies between these two structures may point to modules hidden by a kernel rootkit.

All detection tools were evaluated in a virtualized environment and a contained network. Depending on the available rootkit features (e.g., backdoors), an additional system was added to the network to simulate the attacker. After each rootkit evaluation, the virtual machine was fully reset. Depending on the rootkit’s age, either Ubuntu 22.04.2 LTS or 14.04.6 LTS was used as the operating system. No additional software was installed on the machines besides the rootkit, the detector, and their dependencies. The following sections detail how each tool was set up and under which constraints it was run.

rkhunter was downloaded from its SourceForge repository. Some of rkhunter’s modules utilize other tools like Tripwire or Unhide. These were not used during our evaluation to accurately assess rkhunter’s standalone capabilities. Besides that, rkhunter was run using its default configuration.

chkrootkit was compiled from source. Besides running the chkrootkit main utility, chkdir was also run independently against the system’s root directory (recursively). This produced significant numbers of false positives from inside the /proc directory, prompting us to filter the directory from the output. Because of this, chkdir is, for example, capable of detecting Reptile’s install directory under /reptile, but not Knark’s install directory /proc/knark.

Unhide was also compiled from source. The sub-utilities unhide-linux and unhide-tcp were run at least once using each available detection-mode. Note, that unhide-tcp, contrary to what its name might suggest, also detects hidden User Datagram Protocol (UDP) ports.

AIDE was installed via the apt package manager. The integrity database was generated using AIDE’s default configuration. After installing the rootkits, the new system state was compared to this snapshot. It is important to note that, depending on the time which has passed between generating the database and using it to check for changes, AIDE may produce large outputs, especially on a busy system. Interpreting these outputs and identifying malicious changes does require some experience. We primarily looked for changes to /etc/ld.so.preload, /lib/ld.so, /etc/passwd and /etc/shadow as well as new files which could be associated with known rootkits.

OSSEC was installed in a Server/Agent configuration following the official installation guide. The rootcheck module is run repeatedly in prespecified intervals. It is also executed as part of the agent’s installation procedure. The outputs were viewed on the server’s Web interface.

ClamAV was installed via apt and its malware signatures were updated using freshclam. It was then run recursively against the system’s root directory. ClamAV directly outputs infected files. Out of interest, we also looked into the scanning logs and noticed that in the instance of one rootkit (Father), an error occurred, when ClamAV attempted to scan one of the files manipulated by the rootkit. After some consideration, we decided to not count this as a successful detection, since it is not part of the tool’s regular output and was found coincidentally.

LKRG was compiled from source and its LKM was loaded into the kernel. This automatically generates the integrity database. The outputs of LKRG were read directly from the system’s kernel logs.

A “successful detection” includes all anomalous behavior that is not present on an uninfected system. A detection tool does not necessarily have to identify the threat as a rootkit or by a specific name for the detection to be counted as such. One rootkit (Puszek) caused a few tools to crash when scanning its files. We decided to count this as a successful detection. All utilities were also run at least once on a clean, uninfected system to determine their proneness for outputting false positives.

Some detection methods depend on certain rootkit features being active or previously collected data (see Table 2) being available. To most accurately assess the effectiveness of the tested detection tools, we divided the evaluation into four distinct scenarios based on two criteria:

For this evaluation, “all additional features active” means: If the rootkits implements a backdoor, an active connection is established to it from a separate system. If the rootkits offer process hiding capabilities, at least one hidden process is created. If the rootkit implements arbitrary file hiding, at least one file and one directory is hidden. If the rootkits offers local privilege escalation, at least one process has its privileges elevated using it.

Table 2 shows the previously collected data and knowledge required by the tools for some or all of their detection approaches to function.

As shown in Table 3, the results indicate that separating the evaluation into these scenarios was sensible. Many tools fail if the rootkit has only been installed and not all its features are active. Some are also heavily dependent on proactively obtained data, whether in the form of known file paths, integrity images, or special expertise. Tools like AIDE or rkhunter become almost entirely unusable without such data. Others like LKRG lose a significant number of detections, but can still find some rootkits using alternative means. Approaches exclusively covering kernel space or user space rootkits naturally fail at covering the entirety of the evaluated samples.

Comprehensive evaluation results, covering all scenarios, rootkits, and detection tools, along with the methods used for successful detection, are detailed in the appendix.

All analyzed rootkits implement mechanisms to hide files from the system user. However, these techniques are based on hooking functions, manipulating their output, and removing files from tools that list directory contents. None of the analyzed rootkits actually makes changes to the file data on the drive itself.

Knowing this, an important step in a forensic analysis is to directly parse the file system on the potentially infected drive. This can either be done live by using a tool like the Sleuth Kit or dead by copying the drive and mounting the image read-only on a separate, uninfected system. The latter may not always be possible due to resource and time limitations, but comes with the advantage of being immune to potential rootkit interference. However, none of the rootkits analyzed in this article implements measures against directly parsing the file system on a live system.

The challenge now is to identify the files belonging to the rootkit. This task can be approached in two ways, in addition to a fully manual file-by-file analysis. The first is to search the image for known IoCs. These could come in the form of file paths, signatures, or YARA rules. In theory, preexisting malware solutions might be used for this as well. ClamAV, however, offers insufficient signatures for the tested rootkits. For the 21 rootkits analyzed in this article, we extracted a set of IoCs, which can be used for their detection. Table 4 provides a brief overview; the full dataset is available in the accompanying GitHub repository for this article.

This approach comes with the obvious drawback that one can only find known rootkits. The second approach is more generic and focuses on finding differences between the files on the drive image and those on the live system. Finding a file on the drive that is not present on the live system could be a sign that a rootkit is hiding the file. There might be edge cases, if there is a significant time difference between the drive image acquisition and the comparison with the live state. Figure 1 illustrates an example, in which a directory belonging to the Reptile rootkit is visible in Autopsy, but not when listing the same parent directory on the infected system.

To make the employment of this technique more efficient and feasible in a real-world scenario, we have developed a utility based on The Sleuth Kit³ by Brian Carrier, which automates this process as shown in Listing 4. This tool has already proven its effectiveness by identifying an undisclosed rootkit in a practical setting. It will be made available concurrently with the publication of this article.

Additionally, if the hidden file is a directory, its presence may also be detected using the hard link method utilized by chkrootkit. Counting the directories in Figure 1 (23) and comparing them with the number of hard links pointing to the directory (24) one notices that a directory seems to be missing. However, this method relies on the rootkit not manipulating this number and should be used with caution and only as an additional test. It can also only detect the presence of a hidden directory inside a parent directory, but never its name.

As shown in Table 1, 14 of the analyzed rootkits implement network hiding capabilities. Similarly to the file-hiding functionality of most rootkits, network connection hiding is also based on hooking and manipulating functions on the live system. Therefore, these changes can only affect the output of tools running on the infected system. Software running on a separate network node will be able to see all untainted traffic. Again, the challenge is identifying packets as part of a rootkit’s network traffic. The first approach is to manually analyze the packets and look for suspicious indicators. A rootkit backdoor may, for example, utilize an open Secure Shell (SSH) port but send unencrypted data through it. Figure 2 is an example of such traffic in Wireshark.

In other cases, the packets may contain custom protocols or keywords that may be attributed to known rootkits. This kind of detection, however, falls into the category of general malware/anomaly detection and is not specific to rootkits. The repository linked above still contains a selection of network IoCs for some of the analyzed rootkits to aid this task. The second and more generic approach would be comparing the connections visible on separate network nodes with those visible on the infected system. The difference may reveal connections hidden by a rootkit. Figure 3 shows an open connection belonging to the Father rootkit captured on a MITM proxy, which at the same time is not visible in netstat on the receiving system. Full packet captures are not necessary for this kind of comparison, a Netflow or similar providing metadata is sufficient.

In practice, timing these comparisons can be challenging, especially if the rootkit only communicates for short intervals. Another method to detect hidden connections on a live system is the one already implemented by Unhide and OSSEC/Wazuh. They iterate through all 65,535 possible network ports for the TCP (6) and the UDP (6) and attempt to bind themselves to each of them. If the binding fails, meaning the port is already in use, they check the output of netstat or ss to verify if the connection is visible and if not, report it as potentially hidden. While successful in some cases, this method struggles if a rootkit (e.g., Father) utilizes a constantly listening network service, like an SSH client, since the port will be blocked regardless of whether an active connection has been established or not. Overall, this technique is able to detect connections of 6 out of the 14 rootkits implementing network hiding capabilities.

Eighteen of the analyzed rootkits perform process hiding, and almost all of the analyzed kernel rootkits manipulate kernel structures to either hook functions or hide their modules. Investigators can detect these by analyzing memory images from an infected system.

The first hurdle of memory analysis is to acquire an image of the infected system’s memory. If the system is virtualized, the image can be created from the outside without issues. On a bare-metal system, the image has to be created live using a tool like LiME. None of the analyzed rootkits implement functionality to intercept this process. However, caution is warranted since the data is collected on the infected system. Future rootkits may attempt to exploit this.

Afterward, the image can be analyzed on a separate system using advanced analysis tools such as Volatility. Volatility already offers a multitude of plugins, which are specifically tailored toward detecting anomalies caused by rootkits. The first set of plugins may be used to verify the function pointers of system calls (linux\(\_\)check\(\_\)syscall), interrupt handlers (linux\(\_\)check\(\_\)idt), vfs file operations (linux\(\_\)check\(\_\)fop), and network handler functions (linux\(\_\)check\(\_\)afinfo). If one of them no longer points into the core kernel text, it is marked as being HOOKED. However, this will only detect hooking methods that directly replace function pointers. Rootkits using unconditional jumps, ftrace callbacks, or eBPF hooks will not be detected by these plugins. Jump hooks may be detected by manually inspecting the starting bytes of functions and looking for assembly instructions implementing an unconditional jump to an address belonging to a kernel module. As already mentioned, the custom plugins developed by Case et al. [2] were not available for evaluation. In theory, their approach should be able to detect the three analyzed rootkits utilizing ftrace/eBPF. Six of the analyzed kernel rootkits only hide their modules from /etc/modules to prevent being revealed via lsmod. Nevertheless, since they leave their kobjects (exported via /sys/module) unhidden, they may be detected by comparing the two structures and printing the difference. The linux\(\_\)check\(\_\)modules plugin automates this process, but rootkits that hide themselves from both structures remain undetected. The linux\(\_\)psxview plugin can be used to find hidden processes by comparing different sources for acquiring process lists. This may produce false positives, requiring manual analysis to identify the hidden process. Malicious preloaded libraries may be found using the linux\(\_\)proc\(\_\)maps plugin and looking for unusual shared objects mapped into processes. This approach, however, is entirely based on suspicious names or known strings in the mapped memory segments. The linux\(\_\)netfilter plugin is designed to list Netfilter hooks registered by LKMs. One of the analyzed rootkits uses such a hook as a backdoor trigger. However, during testing, we ran into problems with both the Volatility 2 and Volatility 3 versions of the plugin. The Volatility 2 version was runnable but did not yield any results, while the Volatility 3 version provided in the official community repository contained multiple deprecated code segments. Fixing these was out of the scope of this article. However, since no additional obfuscation is implemented, the plugin should, if updated, be able to find the Netilfter hooks of the analyzed rootkit.

Table 5 shows which of the analyzed rootkits can be detected using the presented Volatility plugins. It is assumed that hidden processes are active and IoCs are available to identify malicious libraries. Without these, the linux\(\_\)psxview and linux\(\_\)proc\(\_\)maps plugins will not produce any results.

Since many of the techniques presented in the previous sections work by comparing the state of certain system structures acquired from different sources and searching for differences, the question arises of how these techniques fare against rootkits that “hide in plain sight” and do not implement significant hiding capabilities. We believe that these cases do not fall into the realm of rootkit but rather regular malware detection, which is an entirely new topic in and of itself and not in the scope of this article.

Comprehensive detection results, covering all rootkits and the primary techniques presented in this chapter, along with the methods used for successful detection, are detailed in the appendix.