Kernels, kernels, everywhere!

I’ve been playing around in the early stages of the boot process. I haven’t fully formed a clear goal, but because I’m interested in poking around the init process, it’s made me want to better understand how and when the kernel hands off to the “actual” OS.

I have no guide or formal training for any of this, so this is mostly written from the perspective of a beginner figuring stuff out. In my case, I hope this is a bit of an edge, since I have absolutely no allegiances to any OS, init system, kernel, services, etc. so my initial goal is to learn as much as possible about the different approaches to modern computer operating systems.

Leaning on Nix to build the Linux kernel

NixOS actually makes certain aspects of building the linux kernel straightforward.

You can just simply write a nix derivation in a file (e.g., default.nix) and then build a system with:

nix-build default.nix

You override the kernel defaults in your derivation with:

...

customKernel = pkgs.linux.override {
  structuredExtraConfig = with pkgs.lib.kernel; {
    # Ensure things we need are here
    USB = yes;
    USB_EHCI_HCD = yes;
    ...
    VIRTIO_NET  = pkgs.lib.mkForce yes;
    VIRTIO_BLK  = pkgs.lib.mkForce yes;
    SCSI_VIRTIO = yes;
  };
};

I’ve seen, developed with, and reverse-engineered a lot of complex, ill-conceived, draconian, confusing, spaghetti build processes; so what a delight this simple “flip some switches, run a single command”. I know it’s come a tremendously long way on the kernel side too, so at some point I’ll probably do a more direct build, but this let me get to the part I was most interested in (the init system) quickly. I need more stuff like that these days.

The limitation of this build process though is that it still really steers you to building the generic catch-all kernel bundled with the OS, which takes about 30 minutes to build (since I guess we’re rebuilding all of the modules too), even on a 16 core machine from a few years ago. For the record, I think that’s good default behavior. But when you’re trying to cut down all unnecessary modules for a stripped-down, custom network-booted machine with no real “operating system” to speak of, bundling all of the generic modules in with every atomic kernel build was cool to see it come alive, but quickly got old.

I couldn’t figure out a way to specify “figure out the bare essentials to try and make this work” while also switching off/on the modules I wanted. I spent about a day on it and had a lot of successes but was already losing time setting up and configuring a dedicated build server so that my poor laptop could take a break.

Nix Kernel/Linux Conclusions

The key elements to booting linux are the bootloader, the kernel, and the initial RAM disk (initrd or similar). When the kernel loads the initrd, we eventually call whatever executable /init is baked in. I was able to pop in a very basic Go binary and get some proper signs of life. It almost felt too easy (Update: it was). Definitely enough to keep going.

Linux momentum is absolutely incredible right now, but with that momentum is starting to come more politics. There’s always politics in software (a remarkable amount, actually), but the amount of money flying around AI right now, incentives can start to get warped. IMO, AI (for now) has no real viable way forward at the datacenter scale without Linux, which puts it right at the center of… a lot.

Between my job and my home, I have no viable path forward at the moment without some flavor of linux to start from. Hm. Makes you think.

FreeBSD

Unix lives on in the form of the various BSD flavors: FreeBSD, OpenBSD, NetBSD, and many others. Getting the same basic kernel+root filesystem up and running based on the BSD kernel was way way harder.

It kind of shouldn’t have been. I thought the documentation was excellent. The handoff between the boot loader (delivered over the network via TFTP) and then to the kernel (delivered via NFS, specifically configured using the root-path field of the DHCP server), which then loads mfsroot is well explained in “advanced networking” chapter. But they fail to dive into how mfsroot works and stop at basically booting into the installer over the network. The root-path requirement additionally feels non-standard and pretty easy to miss.

I’d imagine most folks aren’t dropping into the docs trying to figure out exactly the stuff I’m interested in, so this is all easily forgiven. I definitely missed the NFS delivery as a hard requirement the first time around. This is not the case for the linux kernel which was able to resolve its initrd over the TFTP connection without issue.

FreeBSD also has such strict adherence to the filesystem hierarchy standard that I have lots of respect for it, but coming from Nix, which actively rejects the FHS, this is a big change. I had a hard time separating out in the docs which parts of the FHS were strictly required and where.

Just funny how much steeper the curve was on this one. 12 hours later though I’m booted into my own custom init process running on top of the FreeBSD “GENERIC” kernel.

Building the generic kernel (via make -j 8 buildkernel) with no modificiation took about 8 minutes. The generic kernel is 30MB. There’s 864 additional modules that get built totaling up to ~200MB in my quick inspection? The kernel and the modules end up in the “kernel” directory.

The contents of the NFS share:

/pxeroot/std/boot/
├── defaults/
├── kernel/
├── loader.conf
├── loader.rc
├── lua/
└── mfsroot

The contents of the TFTP share:

/tftpboot
└── pxeboot

Lost a lot of time due to:

  • Attempting UEFI configuration first via the loader_lua.efi path
  • Not understanding that I could use /boot/pxeboot
  • loader.rc, lua, and defaults were missing in early attempts

Conclusions

Boy. Definitely not a quick success on FreeBSD but I’m glad I stuck with it and eventually got there. It’s just a complex web of services to glue together to get it booted over the network. The PXE boot over the network added a lot of complexity but it’s my ultimate target so I’d need to solve it at some point. It’s gonna be rad when it’s doing that onto real metal.

The list of people who actually seem to have good ideas about computing is short, in my opinion. Nearly all of them have some affiliation back to Bell Labs and likes of Ken Thompson, Dennis Richie, Brian Kernighan and all the others who helped contribute to the early days of Unix, from which BSD is derived. The best (factually, accurate, not opinion based) programming language is of course Go, which ties its legacy back to Bell labs via Rob Pike, who also helped give us other tech like UTF-8 encoding.

I think this group deeply understood that computers needed to make life easier, otherwise, why would you be using them? We need to do complicated things, but with less and less complexity on our end. Modern software (including some of the linux conversations, flavors, debates, etc.) is losing this as a core focus.

FreeBSD asks a little bit more of its users (understatement of the year), and now I’m invested. In hindsight, as things often are, the mistakes were simple and understandable and when there are that many it just take a while. I’m also biting off a lot all at once having never done kernel building or development. Feels good though. Learning a lot.

Not many feelings as good as watching that screen actually come alive and start outputting what you told it to. The machine layer has always been the most interesting part. :)

Final Thoughts

Both FreeBSD and NixOS are obtuse in their own ways, but they’re obtuse in opposite directions. To me, NixOS feels like what you have to do because people don’t know how to act right (atomic, repeatable, enforced builds) and FreeBSD is like utopia with perfect adherence to the FHS, but will fall apart the moment you get someone not invested in preserving that (or give them too many permissions). The BSD system is speaking to me a little bit more.