Fuzzing the Linux kernel (x86) entry code, Part 1 of 3

If you’ve been following Linux kernel development or system call fuzzing at all, then you probably know about trinity and syzkaller. These programs have uncovered many kernel bugs over the years. The way they work is basically throwing random system calls at the kernel in the hope that some of them will crash the kernel or otherwise trigger a detectable fault (e.g. a buffer overflow) in the kernel code. While these fuzzers effectively test the system calls themselves (and the code reachable through system calls), one thing they don’t test very well is what happens at the actual transition point between userspace and the kernel. There is more to this boundary than meets the eye; it is written in assembly code and there is a lot of architectural state (CPU state) that must be verified or sanitized before the kernel can safely start executing its C code. This blog post explores how one might go about writing a fuzzer targeting the Linux kernel entry code on x86.