Hey I am Ankan, I have been a linux user for good part of a decade now. What started as an effort to save my aging laptop from obsolescence, turned into an endless series of rabbit holes that got me here. After the ritual distro hopping days, and settling down with a comfortable Artix Linux config I hit a plateau. Things like mesa, device tree, wayland protocol, libc, stack trace, system calls, fork, etc. stuff that I would regularly come across in forums but always remained a bit of a blackbox. I felt like just playful curiosity wasn’t enough to get past these.

System Programming was what I needed but I never could go beyond the first few pages of The Linux Programming Interface and Linux System Programming. So when I found out about Mentorship Program run by the Linux Foundation from r/kernel, I applied right away. With a few weeks to spare before the start I managed to complete the tasks needed to be eligible for it. I was unsure if I would get accepted since I did have to rush through a bit, so I was extremely glad when I found an email invitation waiting for me in my inbox.

In our first meet Shuah welcomed us and provided some initial guidance - to use our legal names, as the contribution make is under the name in the patch and to focus on at most two subsystems. I had a vague idea what a subsystem meant, it is the the highest level of organization, where each subsystem has its maintainer/s and reviewer/s. Establishing proper communication with them was one of the key objectives.

When sending a sending a contribution if we had any problems we could ask on Discord and generally static analysis and syzbot bugs were the recommended place to start for mentees.

The initial challenges for me were:

1. Syzbot bugs beyond my comfort zone.
2. The information in the docs were condensed, I wasn’t extracting much info.
3. Difficultly in keeping track of which header imports what and from where and general inexperience with C.
4. Clever use of macro, took me time to learn and appreciate.
5. Ineffective use of git. Git is so much more than just commit and merge.
6. Scattered efforts, so I was not making much real progress.
7. In addition to this I made a mistake by not going through each mentee blogs and the resources list more rigorously. Most of the problems I encountered and ended up reinventing the wheel were already documented there. I have tried to categorize things in Kernel Resources (should get updated frequently).

After the initial setbacks I decided to get some foundational OS knowledge, Kaiwan N. Billimoria’s Linux Kernel Programming and Hands-on System Programming was how I understood concepts like paging, virtual memory, kernel modules, scheduling and process management, concurrency, interrupts, ftrace/trace-cmd, memory allocation, locking etc. These are really some core things that anyone learning about the kernel would need to understand. The books came with a git repo containing exercises which helped root the concepts. Even when these were quite recent books some of the things had changed, but it was nothing unmanagable.

I actually had expected to cruise through the rest of the mentorship having finished around 1500 pages, but as I progressed I came to the realization that books can only provide the basic foundational concepts, and as a good starting point, the ultimate source of truth remains in the kernel source itself and everything else is just a stepping stone. So navigating around the source shouldn’t have been ommited for the period. I did find a few other books of interests that I would like to get back to - John Madieu’s Linux Kernel Driver Development, Paul E. McKenny’s Perfbook and good ol’ LDD.

With the new understanding I was able go through the docs better and tools like cscope & clangd allowed me to better navigate the source.

With half of the time remaining I decided to start looking for some easy patches, since a minimum of 5 patches is required for the graduation. I resorted to static analysis tools as they seemingly had simpler fixes. The only issue was that there were a lot of many of them and most are ones with no quick fix and something low priority from a maintainer’s perspective.

Apart from a handful, a lot of mentees struggled to get patches accepted, so I was careful not to rush in with my patches. Although one patches did get rejected right away as I was modifying userspace behaviour. A workqueue api cleanup patch that felt was a safe bet had an issue which Khalid Aziz(mentor) pointed out while reviewing; you can never be too careful, it is always a good idea to get the initial patches reviewed before sending. There was a point where I spend one whole week, trying to do a TODO in realtek ethernet driver (r8169) only to figure that my network adapter loaded different firmware than the affected codepath. Thankfully the rest of the patches I sent got merged, these were mostly documentation fixes and one was cleanup. So nothing to brag about yet.

My general understanding came to this:

1. Getting a feature merged in the kernel is challenging if you aren’t sure. Even if you are confident, chances are there would be a long discussion and some changes proposed before it gets accepted.
2. A code cleanup work which doesn’t fix anything or is not being actively pursued by the kernel community may be frowned upon depending on how busy a subsystem is.
3. The general assumption is that patches are expected to have been properly tested. If not, then it needs to be mentioned. If your patch is hardware dependent you need to test it on real hardware yourself or request someone who owns one.
4. Syzbot bug fixes get accepted almost immediately.

Overall I had 6 patches merged, learn’t about Kunit and Kselftest, those turned out to be more beginner friendly and serve as documentation themselves.

Towards the end of the program I had managed to acquire an old laptop with a ryzen 2500U APU for testing purposes. Qemu while convenient for testing syzbot bugs, will always fall short when testing drivers, which needs actual hardware to run & test. The drm subsystem had a great TODO list which I wanted to work on.

With little time I was granted some extension in which I managed to find a regression in amdgpu driver since kernel version 6.13, where on reloading the module it would crash on certain operations. This seems to be hardware timing issue on the gfx IP, so for the time being the drm subsystem is where I will focus. I was able to reproduce the issue with igt-gpu-tools and then with my own reproducer. The older kernels blocks the operations that cause the crash, which could be a valid patch, but I am still pondering if that would be the ideal way to go.

1. Solve a syzbot bug. I am yet to fix one.
2. Follow the discussions on LKML & LWN regularly.
3. Focus on the drm subsystem.
4. Review patches. The kernel has too many developer, not enough reviwers/maintainers.
5. Regular blog posts about low-level stuff.
6. Definitely a career in low-level programming.

The Mentorship Program was an enriching experience, getting to work on the kernel with actual kernel maintainers as mentors! They are indeed very underappreciated heros making sure a billion devices (20 billion actually) dont miss out on those cpu ticks. It also got me to finally set up my my blog, this very blog post is part of the graduation criteria. The kernel is still largely mysterious to me, but instead of going “oh its a kernel issue” and quietly back off, now I can confidently say “let me take a look at it”. It is a very liberating feeling.

Many a thanks to my mentors Shuah Khan, David Hunter, Khalid Aziz for the unforgettable experience, my fellow mentees and ex-mentees from whome I got to learn so much and the maintainers for providing constructive feedback. I hope we get to work again in coming years.

Cheers! :)