Setting up a debugging environment for the Linux Kernel

Whether you are developing a kernel module or doing some kernel hacking, it’s essential that you set up a development and debugging environment which will help you in complete the project faster and with ease. The method of using printk to display debug info is a quite limiting one when it comes to developing more complex modules/drivers. That being said, the reason I’m writing this is to share with you everything I learned from setting up the tools for testing and debugging a LKM rootkit.

When it comes to debugging the Linux kernel there are a few options to choose. If you have available hardware laying around you could use two machines; a development machine connected to the KGDB infrastructure of the kernel running on the target machine via a serial connection as described in here. An other option, more practical, is the use of KDB which does not require additional hardware although has other limitations because it’s not a source level debugger like kgdb but rather a kernel-level debugger. The final and most compelling option in my opinion is visualization, you can run the kernel on a virtual machine with VMware that will run as a userspace application which can be manipulated easily and even if the kernel crashes, the development machine will not be affected. More fitting even than the the VMware or some other visualization application, is a machine emulator and visualizer called QEMU, which gives us the option to test our modules for a range of different architectures because QEMU can emulate different processor architectures. In this article we will describe the process of debugging the kernel with the use if QEMU/KVM.

Next we have to examine our options regarding running the kernel inside QEMU. I see no point running a complete Linux based operating system since we are more interested in the kernel and the modules of that kernel, so a better option is to use tools like Buildroot that is an easy to use tool in order create simple embedded systems. With buildroot we can obtain an image of a kernel version we want to test and a small root filesystem which will contain our Linux Kernel Module we want to test/debug. This is convenient because QEMU can run directly from a kernel image and a rootfs.

Installing and configuring the tools

Having specified the tools that we will need, lets see step by step the process of creating the environment. In this example, we chose to compile kernel version 4.9.6,  target’s architecture x86_64 and development machine is Ubuntu 16.04 x86_64. Firstly, get Buildroot and QEMU.

# get QEMU (debian based operating system)
$ sudo apt-get install qemu
# get Buildroot
$ wget https://buildroot.uclibc.org/downloads/buildroot-2017.02.tar.gz
$ tar xzvf buildroot-2017.02.tar.gz
$ cd buildroot-2017.02

Then we need to configure Buildroot. You can use a default configuration, as shown below, and after edit that configuration to save time

# firstly select the default configuration
$ make qemu_x86_64_defconfig
# use menuconfig to change some params of the default config
$ make menuconfig

Change the following:
• At Kernel version menu item, set the kernel version you would like to compile
• Enter tty1 at Port to run getty (login prompt) on, inside System Configuration menu.
• Optionally, set at Number of jobs to run simultaneously the number of core in your machine, located at Build options menu.

Also, we need to configure the kernel that we will compile with the help of Buildroot

$ make linux-menuconfig

• Inside menu Kernel hacking → Compile-time checks and compiler enable the following, please notice that

kernel hacking

option should be enabled so the following options should be available:
• Compile the kernel with debug info option.
• Provide GDB scripts for kernel debugging option (if the kernel version supports this feature).

Finally, issue a make command that will create the root filesystem and build the kernel image.

$ make

The output of the process above should be stored inside the output/ directory. We are ready to proceed with the debugging.

Debugging the Linux Kernel

Having configured the Buildroot, compiled the kernel and generated the rootfs, it’s time to run the virtual machine with qemu:

$ qemu-system-x86_64 -kernel output/images/bzImage -smp 2 -m 1024 \
-hda output/images/rootfs.ext2 -append "root=/dev/sda rw" -gdb tcp::55666 \
-enable-kvm

Tip : If you want to debug the kernel initialization then append the -S flag, this will stop the execution before the first instruction.

In order to start debugging we must fire up gdb and pass as argument the location of vmlinux which is an object file containing the Linux kernel and it is located inside output/build/linux-(your version). Also we must connect to the target’s gdb stub at port 55666 which is the one we instructed QEMU to use with the command above. Finally, if the target’s architecture is different than our host architecture then we must set up the gdb for that architecture. The command that achieves all of the above is the following:

$ gdb output/build/linux-4.9.6/vmlinux -ex 'set architecture i386:x86-64' \
-ex 'target remote localhost:55666'

Tip: The continue command enables the Linux guest system, and pressing Ctrl+C interrupts execution.

At this point we can start setting breakpoint in kernel functions, examine the registers and kernel variables.
For example, let’s traverse task list which is a linked list of task_structs (more info about task structure here).
We start with init_task, the head of doubly linked list which holds all task_structs in the system and the task_struct for the “swapper” process, and we follow the list task_struct.tasks. Because the pointer ->next of the tasks list points to the next tasks structure inside the task_struct we must find the offset of the tasks member from the task_struct and subtract it from the task_struct->tasks->next to get the pointer to the start of the next process descriptor (task_struct).

# print the name of the init_task (pid 0 and name "swapper")
(gdb) p init_task->pid
$1 = 0
(gdb) p init_task->comm
$2 = "swapper/0\000\000\000\000\000\000"

# find the next task
(gdb) set $offset = &( (struct task_struct *)init_task)->tasks
(gdb) set $next_head_list = init_task->tasks->next
(gdb) set $next_task = (struct task_struct *)( (char *)$next_head_list - (char *)$offset)

# print the next task ( pid 1 and name "init")
(gdb) p $next_task->comm
$3 = "init\000er/0\000\000\000\000\000\000"
(gdb) p $next_task->pid
$4 = 1

Debugging Kernel Modules

To be able to debug Kernel Modules we must compile the module against the kernel built with Buildroot and insert the module inside the root filesystem generated also with Buildroot.
Firstly, set the Kernel Directory variable of the module Makefile to the directory in which Buildroot build the Kernel, that should be at output/build/linux-(version). For example the Makefile of a testing module in our case is the following:

obj-m := test-module.o
KDIR = ~/buildroot/buildroot-2017.02/output/build/linux-4.9.6/
PWD = $(shell pwd)
default:
make -C  $(KDIR) M=$(PWD)
clean:
rm -rf *.o *.ko *.symvers *.mod.* *.order

Secondly, place the object file produced from the compilation of the module inside the rootfs and issue a make command to regenerate the rootfs:

# move or copy the object file to the root filesystem 
$ mv test-module.ko ~/buildroot/buildroot-2017.02/output/target/root
# regenerate the rootfs
$ cd ~/buildroot/buildroot-2017.02
$ make

This process of moving (or coping) the new module object file and the regeneration of the rootfs must be performed after every change we make to the module’s code, so a bash script can save us some time.

After the regeneration of the rootfs reboot the virtual machine and the module object file should be inside /root in the target machine. Load the module with lsmod command inside the target.

# from the target machine, load the module
# insmod test-module.ko

Now we have to fire up gdb like before except this time from the directory we built the kernel because we want to use the Linux-provided gdb helpers that provide Kernel awareness and will help us debug the modules.  Also, because there is a restriction in gdb that will not allow loading of untrusted scripts (scripts not found inside specific directories) we have to explicitly instruct gdb to load scripts from the current directory with the add-auto-load-safe-path command.

$ cd output/build/linux-4.9.6/
$ gdb vmlinux -iex 'add-auto-load-safe-path .' -ex 'set architecture i386:x86-64' -ex 'target remote localhost:55666'

Call the symbols script (lx-symbols command) and pass as an argument the directory in which the module’s object file resides, in this example the test-module.ko file is located under ~/LKMdevelopment directory. As a result the symbol information of any modules that are located inside the directories you specified and are currently loaded in kernel memory, will be added to the gdb and you will be able to debug those modules. You can even set pending breakpoints in modules that are not yet loaded and when are loaded the breakpoint will hit. This allows you to debug the function module_init() or subsequent functions called by module_init(). The process is shown below.

(gdb) lx-symbols ~/LKMdevelopment
(gdb) break dummy_function
Function "dummy_function" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (dummy_function) pending.
(gdb) c
Continuing

# switch to target machine and load the module and go back to gdb again

scanning for modules in /home/ouga/LKMdevelopment/
scanning for modules in /home/ouga/buildroot/buildroot-2017.02/output/build/linux-4.9.6
loading @0xffffffffa0008000: /home/ouga/LKMdevelopment/test-module.ko
[Switching to Thread 2]

Breakpoint 1, init_module () at /home/ouga/LKMdevelopment/test-module.c:20
20        dummy_function();
(gdb)

One thing to notice it that the gdb helper scripts where included in kernel version v4.0-rc1 so you need that or later version, else apply this patch to your kernel.

---

Extra: Debugging modules without the use of lx-symbols script
Alternatively, you may choose to manually tell the debugger where the code and data segments where loaded by viewing the sections through /sys/module/(module-name)/sections/ and then use add-symbol-file command in gdb to load the symbol information for the module as described here. Also in order to debug the module initialization without the help of the script then we need to set a breakpoint at do_init_module() and examine the module structure passed as pointer to the function. Inside the structure we can find core_layout member which is of type module_layout  and contains pointer to the memory address where the module’s code and data is located inside the Kernel. In this way we managed to stop the execution before the module_init() is called and we to locate where the module is loaded, now all that needs to be done is to use add-symbol-file command and pass as argument the memory address of the module.

Breakpoint 1, do_init_module (mod=0xffffffffa00000c0) at kernel/module.c:3367
3367    {
(gdb) p mod->core_layout->base
$1 = (void *) 0xffffffffa0000000
(gdb)add-symbol-file ~/LKMdevelopment/test-module.ko 0xffffffffa0000000
add symbol table from file "/home/ouga/LKMdevelopment/test-module.ko" at
    .text_addr = 0xffffffffa0000000
(y or n) y
Reading symbols from /home/ouga/LKMdevelopment/test-module.ko...done