TLDR: The `kernel-headers` packages in Fedora, RHEL, or CentOS do not carry the files you get by installing `linux-headers-$(uname -r)` on Debian based distros. If you want to build add-on modules for the kernels in Fedora, RHEL, or CentOS, you need to install `kernel-devel` instead. Fun fact: I consider both approaches to package naming flawed.
False friends
Packages in different, unrelated distros sometimes have similar names, but nevertheless contain something entirely different. People that don't keep this in mind are sometimes going to have a bad time.
The two package names are thus something like 'false friends': they look or sound similar, but differ significantly in meaning.
You actually need to install a package called `kernel-devel` to build add-on modules for the kernel from Fedora, RHEL, or CentOS . They also ship a `kernel-headers` package, but don't bother with it, it's not directly needed.
Neither package name is fully accurate
You now might wonder, which of the two approaches in naming is corrent: `linux-headers-foo` or `kernel-devel`? The short answer from my point of view: neither, as both names are a bit off.
To understand why, let's take a closer look at the Linux kernel – that what is being packaged here and thus should be a strong indicator of what's right or wrong. When looking at the sources you'll actually find something that is relevant in this context: a make target called `headers_install`. The help text describes it as 'Install sanitised kernel headers to INSTALL_HDR_PATH'.
That doesn't tell much, so people familiar with `linux-headers` packages might quickly jump to conclusions like 'that will install the files needed for building add-on modules'. But that not the case, as the files installed by this target are something entirely different: the ```kernel headers for use by userspace```. They ```[…] describe the API for user space programs attempting to use kernel services. These kernel header files are used by the system’s C library (such as glibc or uClibc) to define available system calls, as well as constants and structures to be used with these system calls. […]```.
Those quotes are from the Linux kernel documentation. To read them in full context head over to `Documentation/kbuild/headers_install.rst` in the Linux sources or to its rendered version on kernel.org.
The files installed by `make headers_install` are not really relevant when it comes to building kernel modules, but crucial in a totally different situation: when building your system's C library. That's why 'Linux from Scratch (LFS)' installs them right before compiling the GNU C Library (glibc).
The Debian-universe ships the files installed by `make headers_install` in a package called `linux-libc-dev`. The RH-universe on the other hand ships them in a package called `kernel-headers`.
Missing link
So if `make headers_install` does not install the files needed for building add-on modules, what make target will install them? Thing is: the Linux kernel does not offer a make target to do just that. It never has.
The concept of shipping all the files for building add-on modules originated in the distro world somewhere. I don't known when and where exactly. I first saw it in Fedora in its early days (2003? 2004?), when Arjan van de Veen was maintaining the Fedora kernel (IIRC). He simply made the RPM install section collect the files needed for building modules and shipped them in a new sub-package of the kernel package. The latter was and still is shipped in a package called `kernel` and the sub-package got the name `kernel-devel`. That's why you need to install this package when you want to build add-on modules.
That approach followed a familiar pattern in the RH-universe, where development files are always in '-devel' sub-packages. It's thus totally normal for users to install `foo-devel` if they want to build something for or on top of a software shipped in a package named `foo`. It's not that different from the Debian-universe, which uses '-dev' instead.
Choice of names
Other distros also started packaging the files to build add-on kernel modules, but chose other names for the sub-package. The Debian-universe, which ships the Linux kernel in a package called `linux`, settled on packages with the prefix `linux-headers`. I don't known how it came to this or if `linux-dev` was even considered.
Is 'headers' an appropriate suffix, even if `make headers_install` installs something different?
Decide yourself. In Fedora 33 the kernel-devel package currently contains about 16.500 files. About 12.000 of them indeed are header files (their file names end with '.h'). Then there are a bit more than 4100 files called `Makefile` or `Kconfig`, which are needed for configuration and building. Then there are about 80 files containing C code and about 250 other files: some scripts (perl, python, coccinelle), more Kbuild stuff and a few other things.
In other words: there is a bit more than headers in there, but they dominate. Nevertheless I still think it wasn't the best idea to use `linux-headers` as prefix. That's mainly for two reasons: (1) a 'linux-dev' prefix would have been more in line with the approach used by other packages in the Debian-universe; (2) it's confusing users, as the kernel installs something different when running `make headers_install`.
But to be fair on the second point here: I did not dig down into the history books, but I strongly suspect the prefix `linux-headers` was chosen and established before `make headers_install` was introduced; that was in September 2006 with Linux 2.6.18.
But in the end it doesn't matter if I find the term appropriate or not: in the Debian-universe the term `linux-headers` and the purpose of these packages is well established these days. Changing it is likely not worth the trouble. People just need to be made aware they need to install `kernel-devel` on Fedora, RHEL, and CentOS when they would install `linux-headers-$(uname -r)` on Debian, Ubuntu or distros based on them.
 |
| xkcd/Randall Munroe, CC BY-NC 2.5 |
Bonus feature: Fedora is wrong, too
Earlier I mentioned that I consider both `linux-headers` and `kernel-devel` flawed, but only explained what's wrong with the former. Well, it's a bit nitpicking, but the problem with the package name used in the RH-universe is on the other side of the dash. Nobody in the RH-universe would put Firefox in a package `browser` or LibreOffice in a package called `officesuite`. But somehow a kernel named 'Linux' was put in a package called `kernel`.
But whatever, fixing this would likely also create more trouble then it's worth. And it's only going to become a problem if Fedora ever decides to ship other kernels (like GNU Hurd or the FreeBSD kernel) in parallel to Linux (like Debian GNU/kfreeBSD).
Appendix A: Older headers are totally fine
Fun
fact: the
packages `linux-libc-dev` or `kernel-headers` mentioned above (those that contain the files that `make headers_install` installs) sometimes look outdated in
Linux distributions, as they sometimes stay on a older version while way newer
kernels with higher version numbers get installed while updating the
system. Don't worry about it, as there is a reason for this. It's
actually explained in the kernel documentation file referred to above:
```Kernel
headers are backwards compatible, but not forwards compatible. This
means that a program built against a C library using older kernel
headers should run on a newer kernel (although it may not have access to
new features), but a program built against newer kernel headers may not
work on an older kernel.```.
In other words: don't update
`linux-libc-dev` or `kernel-headers` if you want to ensure code you
compile works on older Linux kernel versions. That for example might be the kernel
found on and installed by the installation ISO, which people will use
before they update their systems (which some users never do). But don't
worry too much about this aspect, for most software that won't matter
anyway.
Appendix B: Why did nobody upstream the concept behind `linux-headers`/`kernel-devel`?
Still reading? Seems like you are really curios! Good, because I got something more for you.
I explained `make headers_install` above and mentioned the Linux kernel has no make target to install the files that are shipped in packages like `linux-headers` and `kernel-devel`. I don't known why such a make target is missing upstream, but I suspect it's the usual: nobody set down to clean one approach up and submit it upstream (apart from that approach used by the `make bindeb-pkg` target).
But it's IMHO clearly something that would be useful to have upstream: all big distro have packages like that and create it manually, which is error prone; having such a make target upstream would also be handy for people that compile and install their kernel locally, too.
I more than once considered upstreaming this functionally, but never found the time to do so. Or, to be more precise: I never found the motivation(™). ;-)
For the foreseeable future that won't change. I hope someone else will pick this task up. How about you? I guess it would be a good learning exercise and shouldn't be too hard. Looks to me like you might not even have to know C to implement this! Knowing a bit or two about Makefiles might come in handy, but I guess even that is not needed for someone really motivated!
Closing words: If you spot anything here that you consider false or misleading please let me know; especially when it is in one of those sections about the Debian-universe and its packaging, because that's not really my area of expertise. I'm happy to update this text to fix mistakes or provide additional details if they are important to know in this context.
