Aqua’s Team Nautilus detected an intensive campaign targeting cloud native environments that uses advanced persistent threat (APT) techniques usually leveraged by nation-state threat actors. As part of the campaign, the attackers used two types of rootkits to hide their presence and gain persistence over the victims’ systems. In this blog, we provide a deep analysis of these sophisticated techniques, explain how rootkits work, and how adversaries are using them to attack cloud native environments.
An introduction to rootkits
State-sponsored threat actors are known to develop and use state-of-the-art malware and APT-grade techniques to achieve their goals. For instance, the Drovorub malware developed by Russian intelligence (GRU) to target Linux systems uses the rootkit Drovorub-kernel module to hide itself and other files, directories, and network components. Another example is the Moriya rootkit employed in the TunnelSnake campaign by a Chinese-speaking threat actor to secretly take control of organizations’ networks.
But what is a rootkit? And what is it used for?
The term rootkit is a compounding of the words “root” and “kit”, and it stands for a program that provides continued privileged access to a system while actively hiding its presence. In general, there are two main types of rootkits: user space rootkits, and kernel space rootkits. The former operate in a user space, where they intercept and modify calls made by binaries to libraries, while the latter are more dangerous as they provide the broadest user privileges and can control all system processes.
In cloud native environments, attackers usually execute rootkits on the host to hide malicious processes, and to reduce the chances of detection after they have escaped the containerized environment. Also, rootkits can be used to override functions or binaries in user space (for example, using LD_PRELOAD).
Using rootkits in the wild
In a recent campaign by TeamTNT on few dozen hosts in the wild, they’ve been using advanced techniques that are usually leveraged by APTs. The campaign targets cloud native environments; specifically by running malicious containers. The volume of attacks has been intensive. We have witnessed hundreds of attacks per each infected host, with 20 attacks per day on average, and the campaign is still ongoing.
Daily attack volume (May 2021 – August 2021) The primary axis – the volume of attacks The secondary axis – the number of newly infected hosts per day
In each attack, we have seen the following steps:
- Running a vanilla container image: The adversaries run an Alpine container image.
- Escaping to the host: The attackers mount the host file system to escape the container and gain access to the host.
- Downloading a malicious script: After they have escaped to the host, they write a command in the cron scheduler system that is designed to download and execute a malicious shell script (cronb.sh) from a remote source.
- Loading and executing the malware: The script cronb.sh is the payload, which executes the attack.
To hide their presence, the attackers are using two types of rootkits in this script: a kernel space rootkit, Diamorphine, and a technique that changes binaries, which is considered a user space rootkit.
In this campaign, the attackers gained initial access to the targeted cloud environment and aimed to extend their persistence over the host. The lowest-hanging fruit in this attack is to hijack resources by executing cryptominers. But this would increase the CPU usage of the host and raise the likelihood of being caught by security monitoring solutions. By using rootkits, however, the attackers can conceal the high CPU, which decreases the chances of being detected.
Analysis of the attack flow
Let’s take a closer look at how these attacks are carried out.
Plan A: Loading a kernel space rootkit, Diamorphine
During the attack, the script cronb.sh executes, among other things, these two functions: installdia() and securethesystem(). The screenshot below shows that installdia contains a tar file encrypted in base64, which is the source code of the Diamorphine rootkit. The attackers are also testing the system, trying to cover as many Linux distributions as they can (Ubuntu, Debian, Red Hat, Fedora or CentOS, etc.).
Diamorphine is a loadable kernel module (LKM) rootkit for Linux kernels. It lives inside the kernel space and is designed to obtain higher privileges on processes and hide malicious activities. LKM-based rootkits are very powerful, allowing attackers to do almost anything in the system, for example, modify a process behavior or even terminate a process before it starts. However, LKM rootkits are more challenging to use because attackers need root privileges or CAP_SYS_MODULE capability to load them.
In our case, Diamorphine hides high CPU usage from programs like ps or top that are used to track Linux processes by hooking the getdents(), getdents64() and kill() syscalls.
In the script cronb.sh, Diamorphine is encoded as a variable. The adversaries later decode it into a tar file (dia.tar.gz), which is then extracted to three files (diamorphine.c, diamorphine.h and Makefile). These are the source code of Diamorphine. We compared these files with the ones that appear in the original author’s GitHub repository and found no differences, meaning the attackers didn’t modify Diamorphine in any way. In the next step, the Diamorphine code is compiled and turns into the kernel object diamorphine.ko.
Now that Diamorphine is ready to go, let’s see how the attackers are using it to their advantage and turning to a fallback rootkit in case the Diamorphine build fails.
The second function in the script cronb.sh is securethesystem():
In the first row, the attackers are trying to verify that the user is “root”. Diamorphine is an LKM rootkit, therefore, it will only run with the root user. Assuming the attackers have root privileges, they execute “setup_dia” to load Diamorphine.
As you can see in the code above, if diamorphine.ko (a kernel object, which is the loadable kernel module that contains the malware) exists in a specific path that the attackers set, Diamorphine is loaded with insmod diamorphine.ko (insmod is a command that inserts a kernel module).
Fallback plan: Installing a user space rootkit
If the execution above doesn’t work, the attackers pursue a plan B. Instead of Diamorphine, they deploy a user space rootkit, which resides inside the user space and intermediates the user actions with the kernel. Adversaries frequently use this type of rootkits to replace executables and system libraries to modify the behavior of processes and display false information.
In our case, the attackers are using it to alter the programs top, ps, and pstree. For example, in the screenshot above they are replacing the name of the binary ps with ps.original. Next, the attackers add to the same library a new file with the same original name (ps) and an exception not to display ext4 or scan (ps.original $@ | grep -v “ext4|scan”). This way, if the end user calls ps, he won’t see all ext4 related processes.
But you might ask, why is ext4 so important? The screenshots below shed some light on this.
It turns out that under ext4 the attackers are hiding a cryptominer (XMRIG) and its configuration, among other malicious elements in the code.
ext4 is a name of a filesystem. The threat actors are also camouflaging it as a systemd service, so it will look benign, and the rootkit will be able to easily hide all related malicious items under ext4.
Analysis of the Diamorphine rootkit
A deeper dive into the source code of Diamorphine reveals more about its methods. As we already said, the Diamorphine rootkit operates in the kernel space. We will focus on the two core techniques it uses to accomplish its goals:
Hooking syscalls and internal functions
This rootkit uses one of the most popular techniques of kernel manipulation. Specifically, it changes program flows by hooking functions that the kernel uses. In this case, the malware is loading and modifying the syscall table that holds the addresses of syscalls. In the screenshot below, you can see the function that loads the syscall table.
And in the snippet below, you can see the section in the code where this function is called:
After Diamorphine successfully loads the syscall table, it can manipulate some of the syscalls so the addresses will point to a custom-made function rather than the original one. The original addresses of these syscalls are saved aside:
Now Diamorphine defines a new function that will replace the getdents:
And finally, the addresses of these two are switched, so when a new user space process calls a syscall (i.e., getdents), the tweaked syscall table will point to the new function (i.e., hacked_getdents). In our case, Diamorphine targets three syscalls: getdents, getdents64, and kill.
The question is, why would anyone want to hook to getdents and getdents64? The answer can be found in this wonderful blog by TheXcellerator. In a nutshell, nearly all userspace tools that provide information about processes read the contents of the /proc/ filesystem. So, when you call programs such as ps or top that reflect information about running processes, they use the getdents or getdents64 function to enumerate a specific pseudo filesystem (/proc). This rootkit (and basically most rootkits) hides the processes by hooking these functions and changing the returned values by removing the information related to the processes that they want to hide.
Direct kernel object manipulation (DKOM):
Diamorphine is a rootkit that changes program behavior by modifying kernel data structures. Among other things, it hides itself from lsmod, a command that lists all loaded modules. When Diamorphine loads itself, it modifies the modules linked list inside the kernel space, so it could stay hidden:
Also, the DKOM technique can be found in the function give root, which is used to elevate user privileges:
Summary and mitigation
In this blog, we analyzed how attackers have been using APT-grade techniques – rootkits – as part of the campaign targeting cloud native environments. The adversaries used the kernel space rootkit Diamorphine to conceal their attack and a user space rootkit as a fallback in case the Diamorphine malware execution won’t work.
Below are some steps practitioners can take to mitigate the risks of falling victim to such attacks:
Apply Linux updates
As recommended in the joint report by the NSA and FBI “Russian GRU 85th GTsSS Deploys Previously Undisclosed Drovorub Malware”, system administrators should continually check for and run the latest version of vendor-supplied software for computer systems to take advantage of software advancements and the latest security detection and mitigation safeguards. System administrators should update to Linux Kernel 3.7 or later to take full advantage of kernel signing enforcement.
Prevent untrusted kernel modules
System owners are advised to configure systems to load only modules with a valid digital signature, making it more difficult for an actor to introduce a malicious kernel module into the system. An adversary could use a malicious kernel module to control the system, hide, or persist across reboots (National Security Agency).
Use Tracee to detect kernel manipulations
Tracee is an open-source runtime security and forensics tool for Linux, built to address common Linux security issues. Tracee 0.5.0 and above enable practitioners and security researchers to write Rego signatures on top of the Tracee eBPF data collection tool. Aqua also added some basic signatures such as “Attempt to load a kernel module detection”, which allow practitioners to detect the loading of a suspicious kernel driver. For a more in-depth analysis on how you can detect privilege escalation with Tracee, check out the talk by Team Nautilus at BlackHat 2021.
Similar capabilities can be found in Aqua’s solution for organizations, Aqua Dynamic Threat Analysis (DTA), which dynamically analyzes the container behavior in a sandbox to identify attack vectors that wouldn’t be detected with static code scanning.
Detect user space rootkits with Aqua Enforcers
File integrity monitoring (FIM) is a process that validates the integrity of the operating system and application files using a verification method between the current file state and a known, good baseline. One of the rootkits described above included a technique that altered the names of some binaries and inserted new file with the names of these binaries. By using Aqua’s FIM feature, organizations can detect this change in the files and stay alert to such activity in their environments.
This blog was co-authored with Itamar Maouda, Security Researcher at Team Nautilus, Aqua’s research team. Itamar focuses on researching malware and threats in cloud native environments.
