Package names can be false friends, too, as (linux|kernel)-headers show

TLDR: The `kernel-headers` packages in Fedora, RHEL, or CentOS do not carry the files you get by installing `linux-headers-$(uname -r)` on Debian based distros. If you want to build add-on modules for the kernels in Fedora, RHEL, or CentOS, you need to install `kernel-devel` instead. Fun fact: I consider both approaches to package naming flawed.

False friends

Packages in different, unrelated distros sometimes have similar names, but  nevertheless contain something entirely different. People that don't keep this in mind are sometimes going to have a bad time. 

There is one area where this afaics causes trouble quite often: when building add-on modules for your distro's kernel. On Debian or Ubuntu quite a few people learned: before starting the compiler, one has to install additional files by running a command like `apt-get install linux-headers-$(uname -r)`. That's why it's easy to assume the `kernel-headers` package fulfils the same job in Fedora, RHEL, or CentOS. But it's a wrong assumption and will lead to failure, because `kernel-headers` packages in the RH-universe contains something totally different.

The two package names are thus something like 'false friends': they look or sound similar, but differ significantly in meaning.

You actually need to install a package called `kernel-devel` to build add-on modules for the kernel from Fedora, RHEL, or CentOS . They also ship a `kernel-headers` package, but don't bother with it, it's not directly needed.

Neither package name is fully accurate

You now might wonder, which of the two approaches in naming is corrent: `linux-headers-foo` or `kernel-devel`? The short answer from my point of view: neither, as both names are a bit off.

To understand why, let's take a closer look at the Linux kernel – that what is being packaged here and thus should be a strong indicator of what's right or wrong. When looking at the sources you'll actually find something that is relevant in this context: a make target called `headers_install`. The help text describes it as 'Install sanitised kernel headers to INSTALL_HDR_PATH'.

That doesn't tell much, so people familiar with `linux-headers` packages might quickly jump to conclusions like 'that will install the files needed for building add-on modules'. But that not the case, as the files installed by this target are something entirely different: the ```kernel headers for use by userspace```. They ```[…] describe the API for user space programs attempting to use kernel services. These kernel header files are used by the system’s C library (such as glibc or uClibc) to define available system calls, as well as constants and structures to be used with these system calls. […]```.

Those quotes are from the Linux kernel documentation. To read them in full context head over to `Documentation/kbuild/headers_install.rst` in the Linux sources or to its rendered version on kernel.org.

The files installed by `make headers_install` are not really relevant when it comes to building kernel modules, but crucial in a totally different situation: when building your system's C library. That's why 'Linux from Scratch (LFS)' installs them right before compiling the GNU C Library (glibc).

The Debian-universe ships the files installed by `make headers_install` in a package called `linux-libc-dev`. The RH-universe on the other hand ships them in a package called `kernel-headers`.

Missing link

So if `make headers_install` does not install the files needed for building add-on modules, what make target will install them? Thing is: the Linux kernel does not offer a make target to do just that. It never has.

The concept of shipping all the files for building add-on modules originated in the distro world somewhere. I don't known when and where exactly. I first saw it in Fedora in its early days (2003? 2004?), when Arjan van de Veen was maintaining the Fedora kernel (IIRC). He simply made the RPM install section collect the files needed for building modules and shipped them in a new sub-package of the kernel package. The latter was and still is shipped in a package called `kernel` and the sub-package got the name `kernel-devel`. That's why you need to install this package when you want to build add-on modules.

That approach followed a familiar pattern in the RH-universe, where development files are always in '-devel' sub-packages. It's thus totally normal for users to install `foo-devel` if they want to build something for or on top of a software shipped in a package named `foo`. It's not that different from the Debian-universe, which uses '-dev' instead.

Choice of names

Other distros also started packaging the files to build add-on kernel modules, but chose other names for the sub-package. The Debian-universe, which ships the Linux kernel in a package called `linux`, settled on packages with the prefix `linux-headers`. I don't known how it came to this or if `linux-dev` was even considered.

Is 'headers' an appropriate suffix, even if `make headers_install` installs something different? 

Decide yourself. In Fedora 33 the kernel-devel package currently contains about 16.500 files. About 12.000 of them indeed are header files (their file names end with '.h'). Then there are a bit more than 4100 files called `Makefile` or `Kconfig`, which are needed for configuration and building. Then there are about 80 files containing C code and about 250 other files: some scripts (perl, python, coccinelle), more Kbuild stuff and a few other things. 

In other words: there is a bit more than headers in there, but they dominate. Nevertheless I still think it wasn't the best idea to use `linux-headers` as prefix. That's mainly for two reasons: (1) a 'linux-dev' prefix would have been more in line with the approach used by other packages in the Debian-universe; (2) it's confusing users, as the kernel installs something different when running `make headers_install`. 

But to be fair on the second point here: I did not dig down into the history books, but I strongly suspect the prefix `linux-headers` was chosen and established before `make headers_install` was introduced; that was in September 2006 with Linux 2.6.18.

But in the end it doesn't matter if I find the term appropriate or not: in the Debian-universe the term `linux-headers` and the purpose of these packages is well established these days. Changing it is likely not worth the trouble. People just need to be made aware they need to install `kernel-devel` on Fedora, RHEL, and CentOS when they would install `linux-headers-$(uname -r)` on Debian, Ubuntu or distros based on them.

xkcd/Randall Munroe, CC BY-NC 2.5

Bonus feature: Fedora is wrong, too

Earlier I mentioned that I consider both `linux-headers` and `kernel-devel` flawed, but only explained what's wrong with the former. Well, it's a bit nitpicking, but the problem with the package name used in the RH-universe is on the other side of the dash. Nobody in the RH-universe would put Firefox in a package `browser` or LibreOffice in a package called `officesuite`. But somehow a kernel named 'Linux' was put in a package called `kernel`.

But whatever, fixing this would likely also create more trouble then it's worth. And it's only going to become a problem if Fedora ever decides to ship other kernels (like GNU Hurd or the FreeBSD kernel) in parallel to Linux (like Debian GNU/kfreeBSD).

Appendix A: Older headers are totally fine

Fun fact: the packages `linux-libc-dev` or `kernel-headers` mentioned above (those that contain the files that `make headers_install` installs) sometimes look outdated in Linux distributions, as they sometimes stay on a older version while way newer kernels with higher version numbers get installed while updating the system. Don't worry about it, as there is a reason for this. It's actually explained in the kernel documentation file referred to above:

```Kernel headers are backwards compatible, but not forwards compatible. This means that a program built against a C library using older kernel headers should run on a newer kernel (although it may not have access to new features), but a program built against newer kernel headers may not work on an older kernel.```.

In other words: don't update `linux-libc-dev` or `kernel-headers` if you want to ensure code you compile works on older Linux kernel versions. That for example might be the kernel found on and installed by the installation ISO, which people will use before they update their systems (which some users never do). But don't worry too much about this aspect, for most software that won't matter anyway.

Appendix B: Why did nobody upstream the concept behind `linux-headers`/`kernel-devel`?

Still reading? Seems like you are really curios! Good, because I got something more for you.

I explained `make headers_install` above and mentioned the Linux kernel has no make target to install the files that are shipped in packages like `linux-headers` and `kernel-devel`. I don't known why such a make target is missing upstream, but I suspect it's the usual: nobody set down to clean one approach up and submit it upstream (apart from that approach used by the `make bindeb-pkg` target).

But it's IMHO clearly something that would be useful to have upstream: all big distro have packages like that and create it manually, which is error prone; having such a make target upstream would also be handy for people that compile and install their kernel locally, too.

I more than once considered upstreaming this functionally, but never found the time to do so. Or, to be more precise: I never found the motivation(™). ;-)

For the foreseeable future that won't change. I hope someone else will pick this task up. How about you? I guess it would be a good learning exercise and shouldn't be too hard. Looks to me like you might not even have to know C to implement this! Knowing a bit or two about Makefiles might come in handy, but I guess even that is not needed for someone really motivated!

Closing words: If you spot anything here that you consider false or misleading please let me know; especially when it is in one of those sections about the Debian-universe and its packaging, because that's not really my area of expertise. I'm happy to update this text to fix mistakes or provide additional details if they are important to know in this context.

Fedora 11, kernel-PAE, and what it means for your x86-32 system

There is one small change in Fedora 11 that I guess will confuse Fedora and RPM Fusion users with x86-32 (aka i386/ix86) systems quite a lot, but afaics did not get enough attention yet:

• By default, the PAE kernel will be used on 32-bit hardware, where appropriate(¹).

"Appropriate" is not really explained -- maybe because it's a bit hard to sum up without going into the boring details. But basically it boils down to: The kernel with PAE support will be installed by Fedora 11 for x86-32 on the majority of x86-compatible systems that have been manufactured in the past three or four years(¹). So likely on your system as well if you are running a x86-32 distro on a modern system.

The important part: the package containing the kernel with PAE support is not called "kernel" -- it's called "kernel-PAE" instead. And that's not the only package where "-PAE" is used as suffix. That has major certain consequences on systems where Fedora 11 installs kernel-PAE:

• when you build kernel modules with akmods, dkms, or manually, then you from now on need to install kernel-PAE-devel instead of kernel-devel
• similar for kernel-modules: instead of "yum install kmod-nvidia" your now need to type "yum install kmod-nvidia-PAE". Yum otherwise will try to install the matching kernel without PAE support for you, which (in short without the boring details) is something you most of the time don't want(²).

In other words: the change in Fedora 11 makes lots of howtos, FAQs, articles on the net and in computer magazines confusing, wrong, misleading or harmful (depending on view and specific howto/FAQ/article), because most of those docs don't consider the above fact (yet).

And that's not Fedora's fault -- PAE kernels are around for a long time in Fedora already. But they were used only on a minority of systems. Most (not all!) of those that have written today's howtos, FAQs or articles were likely either not aware of it or chose to ignored it to keep things simple.

That's backfiring now. So go and spread the news on mailing lists, forums and other places where it might be of interest. Feel free to copy-n-paste this whole text or simply point to this blog entry. Thanks in advance!

(¹) E.g. since processors with NX bit became mainstream; NX stands for NoeXecute and is also called Enhanced Virus Protection by AMD and xD-Bit by Intel

(²) not sure, but maybe the yum-plugin "yum-fedorakmod" could have made yum to the right thing and install the proper kmod for the PAE kernel. I never tried and it doesn't matter much as the plugin is not available in the Fedora or RPM Fusion repositories for F11. If someone wants take care of yum-fedorakmod and wants to get it into RPM Fusion then please drop me a line.

Lots of changes

$ for i in 20 21 22 23; do echo -n "2.6.${i} -> 2.6.$((${i}+1)): "; git diff v2.6.${i}..v2.6.$((${i}+1)) | diffstat | grep ' files changed, '; done

2.6.20 -> 2.6.21: 6568 files changed, 319232 insertions(+), 175247 deletions(-)
2.6.21 -> 2.6.22: 7620 files changed, 519591 insertions(+), 266699 deletions(-)
2.6.22 -> 2.6.23: 7203 files changed, 406268 insertions(+), 339071 deletions(-)
2.6.23 -> 2.6.24: 10209 files changed, 776107 insertions(+), 483031 deletions(-)

I like the current kernel development scheme; it might look a bit scary (especially for 2.6.24), but afaics it's way better then what we had in the past when new stuff (sometimes including drivers for new hardware) lingered around in 2.3 or 2.5. way to long.