Circumventing inotify Watchdogs

About The Project

Recently I’ve been building rudimentary file monitoring tools to get better at Golang, and build faux-watchdog programs for research at Arch Cloud Labs. Through this experimentation, I’ve identified some interesting gaps in the inotify subsystem that are new to me, but are well documented in the Linux man pages. This blog post will explore how to circumvent read detections implemented by inotify.

Inotify As a Monitoring Solution

Per the Linux man page, the inotify subsystem:

provides a mechanism for monitoring filesystem events. Inotify can be used to monitor individual files, or to monitor directories. When a directory is monitored, inotify will return events for the directory itself, and for files inside the directory. - https://man7.org/linux/man-pages/man7/inotify.7.html

Excellent, a standard API for the Linux kernel to monitor for events that one may find interesting on a file system. However, the inotifiy subsystem can not inform you of what process made the underlying call. The man page goes on to say:

The inotify API provides no information about the user or process that triggered the inotify event. In particular, there is no easy way for a process that is monitoring events via inotify to distinguish events that it triggers itself from those that are triggered by other processes - https://man7.org/linux/man-pages/man7/inotify.7.html

While the telemetry of inotify won’t tell you what accessed a file, combined with other monitoring sources (such as eBPF) on a Linux system may provide you the telemetry needed. Next, Lets examine leveraging some default inotify tools that could be incorporated into a simple script to gain familiarity with this API.

Monitoring Reads

The inotify-tools package contains utilities that interact with the inotify subsystem so you can quickly build monitoring applications without having to write C code directly. First, lets leverage inotifywait to monitor for any accesses to /etc/passwd. The command below will print to stdout anytime /etc/passwd is accessed along with a corresponding date format.

$> inotifywait -m --format "%T %w" --timefmt "%T" -e access /etc/passwd

The figure below shows inotify triggering on numerous file reads of /etc/passwd and the highlighted lines show the inotify-tools detecting said read.

Excellent, our simple one-liner works as expected. We can monitor all of the things, right? Not quite.

Referencing the man page again, a curious reader will discover a large gap in the ability of inotify to monitor “all of the things”:

The inotify API does not report file accesses and modifications that may occur because of mmap(2), msync(2), and munmap(2).

This set of syscalls provide a way to alter memory pages on Linux. Put another way, these functions can be used to read from files with the caveat that a file descriptor is already open. This is an important caveat, but not a deal breaker as monitoring all open syscalls is unrealistic for any modern Linux system given the mass of data this would result in. While not infeasible for a single file, it’s unlikely this would be something monitored system wide.

Now, let’s explore of a proof-of-concept of implementing said circumvention.

Circumventing Read Detection

First, we’ll open the file to obtain a file descriptor to said file.

int fd = open("/etc/passwd", O_RDONLY);

Next, we’ll leverage mmap to read in data pointed to it via a file descriptor (fd). Remember this is a syscall not tracked by inotify to perform our “reading” and map data pointed to byfin. For a more mature example, one can leverate the stat syscalls to get file size, but here the example is just reading in 512 bytes.

char *fin = (char *) mmap (0, 512, PROT_READ, MAP_PRIVATE, fd, 0);

Finally, data can be printed to stdout via:

printf("%s", fin);

And that’s it! A less than 20 line C file can result in watchdog circumvention. The entire C source can be found below.

#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <fcntl.h>

int main(int argc, char *argv[]) {

    int fd = open("/etc/passwd", O_RDONLY); // will trigger an open event, but we need a handle to the file.

    // https://stackoverflow.com/questions/20460670/reading-a-file-to-string-with-mmap
    // Allocating memory, and read data into from the open file descriptor fd.
    char *fin = (char *) mmap (0, 512, PROT_READ, MAP_PRIVATE, fd, 0);

    // print data to stdout.
    printf("%s", fin);

    return 0;
}

Now re-executing our inotify-monitor one-liner and then executing our “sneaky read” application will result in no detections being generated for the time it was executed!

Interesting Behavior with Golang’s fsnotify

Aside from circumventing read detection, the notify subsystem has an interesting bug (feature?) related to deleting files. The Golang fsnotify package documentation contains an interesting line:

When a file is removed a REMOVE event won’t be emitted until all file descriptors are closed; it will emit a CHMOD instead - https://pkg.go.dev/github.com/fsnotify/fsnotify#section-readme

This is interesting as if one was monitoring for chmod events, you could preemptively capture data about a file prior to it being deleted. Building a simple watchdog application and monitoring /tmp/ shows this behavior by creating and deleting a file within /tmp/ called tmp via the touch command. The image below shows the chmod events being triggered every time rm /tmp/tmp is being executed in a separate terminal.

However, this behavior I could not recreate for files greater than zero bytes in size. Echoing “hello world” to a file in /tmp/ would not result in chmodbeing generated.

Beyond The Blog

Inotify presents limitations for those seeking to use it for a security related solution. However, for simply modifying for new file creations it is sufficient. This would be applicable in situations where you await for new files to get dropped into a directory to then kick off a batch processing job. For more security related monitoring, using something like eBPF via bpftracecan create useful one-liners to execute arbitrary commands when a given syscall is observed.

Thanks for reading!