Using LSM Hooks with Tracee to Overcome Gaps with Syscall Tracing

Tracee is an open source runtime security and forensics tool for Linux, built to address common Linux security issues. By leveraging the advantages of Linux extended Berkeley Packet Filter (eBPF) technology to trace systems and applications at runtime, Tracee analyzes collected events to detect suspicious behavioral patterns. In this blog, I’ll share the lessons we learned from using eBPF and Linux Security Module (LSM) hooks to overcome Linux vulnerabilities such as Time-Of-Check to Time-Of-Use (TOCTOU).

Boosting Security by Design with LSM Hooks

Running applications usually interact with the operating system via system calls (syscalls). For this reason, security practitioners commonly analyze the behavior of the running application via syscalls. It is important to carefully choose the method for collecting syscalls, because there might be a gap between the argument values that are captured and what is actually being used by the operating system.

Using eBPF allows the running of sandboxed programs in the Linux kernel without changing the kernel source code or loading kernel modules (you can learn more about eBPF and its advantages in this blog post). By attaching such programs to Linux Security Module (LSM) hooks using kprobes, we can collect the argument values that were actually used by the kernel. Below are some examples of issues we overcame with LSM hooks:

Making Sense of Relative Paths

In Linux, when reading information from a user program, the path argument can contain a relative path.
For instance, when invoking a program, Tracee can pick up a syscall with the following arguments:

• open("../../directory/file", O_RDONLY)
• open("./test/../../directory/file", O_RDONLY)

The path in these syscalls may refer to the same location, but the problem is that, based on the given arguments, it is difficult to precisely locate the canonical path that is absolute and always unique. LSM hooks, however, allow Tracee to fetch the canonical path from the root of these files. The significance of canonical paths becomes even more clear when building a security policy because a relative path could be too ambiguous as to the intent and effect. Tracee LSM hooks events allow you to rely on a canonical path and easily apply clear, effective security policies.

Figuring Out Linked Files

Argument values that were given by a user program and might reference a file, for example, can sometimes contain links to other files. In Linux, we can create a symbolic link to a file in a manner that one file serves as a reference to another file or directory. The operating system will use the symbolic link to reach the linked file and apply the given command. Here’s an example where we created a file with a benign name python and linked it to a malicious binary my_malware:

When executing python, we can see that two events are tracked by Tracee. The first event is a syscall with the argument value ./python. But in reality python is symbolically linked to ~/bin/my_malware, and thankfully the security_bprm_check event captured by Tracee returns a pathname to the file that was actually executed.

TIC TOC TOU

When attempting to capture the argument values of a user program, if one is analyzing just the syscall arguments, the results can be subject to race condition attack and miss key details. This is because, after the information is read, the user program can change the syscall arguments based on another simultaneous thread.

For instance, when invoking a program, Tracee can pick up a syscall with the following arguments:
execve("/bin/ls", NULL, 0)

A time window opens between when one of the processes’ threads calls the syscall and when it is executed by the kernel. In the beginning of the time window, the pathname argument is passed to the kernel by using a pointer that points to a memory location in the process address space. During this time window, another thread in this program can quickly change the pathname, and this new pathname will be executed by the kernel.

Continuing our example, the other thread can change the first argument from /bin/ls to /bin/malicious, the latter will be executed by the kernel, while the former will be recorded by Tracee. This scenario is widely known and well-documented as TOCTOU race condition. Attackers can exploit it to influence the recorded value between the check and the use. This may lead to inaccurate data collection and mislead security researchers or automated detection tools that rely on Tracee’s data.

This is the primary reason why we decided to use LSM hooks in Tracee along with the syscall data. Events such as security_file_open contain the pathname that was actually used by the kernel and is cross-referenced with the reported regular syscall event.

Making the Analysis Process Easy

As you may know, in Linux almost everything is a file, and to interact with it you need to use a file descriptor. Upon opening a file (e.g. using open), you receive a file descriptor. The following syscalls may use this file descriptor (for instance, openat, unlinkat, execveat, accept, connect, bind, listen, etc.) to interact with the open file. For this reason, if you want to analyze the actions done on a file, you must keep track of its open file descriptors. A UNIX socket allows you to pass an open file descriptor between different programs, which might make the analysis even harder.

With Tracee’s LSM hook events, tracking file descriptors becomes irrelevant, as the LSM events already contain the relevant data such as the full pathname.

Conclusion

Tracee is an easy-to-use Linux runtime security and forensics tool. It is powered by eBPF technology to better understand the runtime behavior of a program and overcome problems that may be challenging to similar security software.

Learn more about Uncovering Malware Payload Executions Automatically with Tracee and other powerful features in our presentation of Tracee at BlackHat

As a best practice, whenever we use Tracee in our products as a runtime security and forensics tool, we complement it with LSM hooks, and recommend that all Tracee users do so as well. Naturally, there are some syscalls where LSM hooks are not available, but so far the following LSM hooks are covered:

If you are already a Tracee fan, keep an eye on changes and new features on the project’s GitHub page.