Package names can be false friends, too, as (linux|kernel)-headers show

TLDR: The `kernel-headers` packages in Fedora, RHEL, or CentOS do not carry the files you get in Debian based distros by installing the package `linux-headers-$(uname -r)`. If you want to build add-on modules for the kernels in Fedora, RHEL, or CentOS, you need to install `kernel-devel` instead. Fun fact: I consider both approaches to package naming flawed.

False friends

Packages in different, unrelated distros sometimes have similar names, but  nevertheless contain something entirely different. People that don't keep this in mind are sometimes going to have a bad time. 

There is one area where this afaics causes trouble quite often: when building add-on modules for your distro's kernel. On Debian or Ubuntu quite a few people learned: before starting the compiler, one has to install additional files by running a command like `apt-get install linux-headers-$(uname -r)`. That's why it's easy to assume the `kernel-headers` package fulfils the same job in Fedora, RHEL, or CentOS. But it's a wrong assumption and will lead to failure, because `kernel-headers` packages in the RH-universe contains something totally different.

The two package names are thus something like 'false friends': they look or sound similar, but differ significantly in meaning.

You actually need to install a package called `kernel-devel` to build add-on modules for the kernel from Fedora, RHEL, or CentOS . They also ship a `kernel-headers` package, but don't bother with it, it's not directly needed.

Neither package name is fully accurate

You now might wonder, which of the two approaches in naming is corrent: `linux-headers-foo` or `kernel-devel`? The short answer from my point of view: neither, as both names are a bit off.

To understand why, let's take a closer look at the Linux kernel – that what is being packaged here and thus should be a strong indicator of what's right or wrong. When looking at the sources you'll actually find something that is relevant in this context: a make target called `headers_install`. The help text describes it as 'Install sanitised kernel headers to INSTALL_HDR_PATH'.

That doesn't tell much, so people familiar with `linux-headers` packages might quickly jump to conclusions like 'that will install the files needed for building add-on modules'. But that not the case, as the files installed by this target are something entirely different: the ```kernel headers for use by userspace```. They ```[…] describe the API for user space programs attempting to use kernel services. These kernel header files are used by the system’s C library (such as glibc or uClibc) to define available system calls, as well as constants and structures to be used with these system calls. […]```.

Those quotes are from the Linux kernel documentation. To read them in full context head over to `Documentation/kbuild/headers_install.rst` in the Linux sources or to its rendered version on kernel.org.

The files installed by `make headers_install` are not really relevant when it comes to building kernel modules, but crucial in a totally different situation: when building your system's C library. That's why 'Linux from Scratch (LFS)' installs them right before compiling the GNU C Library (glibc).

The Debian-universe ships the files installed by `make headers_install` in a package called `linux-libc-dev`. The RH-universe on the other hand ships them in a package called `kernel-headers`.

Missing link

So if `make headers_install` does not install the files needed for building add-on modules, what make target will install them? Thing is: the Linux kernel does not offer a make target to do just that. It never has.

The concept of shipping all the files for building add-on modules originated in the distro world somewhere. I don't known when and where exactly. I first saw it in Fedora in its early days (2003? 2004?), when Arjan van de Veen was maintaining the Fedora kernel (IIRC). He simply made the RPM install section collect the files needed for building modules and shipped them in a new sub-package of the kernel package. The latter was and still is shipped in a package called `kernel` and the sub-package got the name `kernel-devel`. That's why you need to install this package when you want to build add-on modules.

That approach followed a familiar pattern in the RH-universe, where development files are always in '-devel' sub-packages. It's thus totally normal for users to install `foo-devel` if they want to build something for or on top of a software shipped in a package named `foo`. It's not that different from the Debian-universe, which uses '-dev' instead.

Choice of names

Other distros also started packaging the files to build add-on kernel modules, but chose other names for the sub-package. The Debian-universe, which ships the Linux kernel in a package called `linux`, settled on packages with the prefix `linux-headers`. I don't known how it came to this or if `linux-dev` was even considered.

Is 'headers' an appropriate suffix, even if `make headers_install` installs something different? 

Decide yourself. In Fedora 33 the kernel-devel package currently contains about 16.500 files. About 12.000 of them indeed are header files (their file names end with '.h'). Then there are a bit more than 4100 files called `Makefile` or `Kconfig`, which are needed for configuration and building. Then there are about 80 files containing C code and about 250 other files: some scripts (perl, python, coccinelle), more Kbuild stuff and a few other things. 

In other words: there is a bit more than headers in there, but they dominate. Nevertheless I still think it wasn't the best idea to use `linux-headers` as prefix. That's mainly for two reasons: (1) a 'linux-dev' prefix would have been more in line with the approach used by other packages in the Debian-universe; (2) it's confusing users, as the kernel installs something different when running `make headers_install`. 

But to be fair on the second point here: I did not dig down into the history books, but I strongly suspect the prefix `linux-headers` was chosen and established before `make headers_install` was introduced; that was in September 2006 with Linux 2.6.18.

But in the end it doesn't matter if I find the term appropriate or not: in the Debian-universe the term `linux-headers` and the purpose of these packages is well established these days. Changing it is likely not worth the trouble. People just need to be made aware they need to install `kernel-devel` on Fedora, RHEL, and CentOS when they would install `linux-headers-$(uname -r)` on Debian, Ubuntu or distros based on them.

xkcd/Randall Munroe, CC BY-NC 2.5

Bonus feature: Fedora is wrong, too

Earlier I mentioned that I consider both `linux-headers` and `kernel-devel` flawed, but only explained what's wrong with the former. Well, it's a bit nitpicking, but the problem with the package name used in the RH-universe is on the other side of the dash. Nobody in the RH-universe would put Firefox in a package `browser` or LibreOffice in a package called `officesuite`. But somehow a kernel named 'Linux' was put in a package called `kernel`.

But whatever, fixing this would likely also create more trouble then it's worth. And it's only going to become a problem if Fedora ever decides to ship other kernels (like GNU Hurd or the FreeBSD kernel) in parallel to Linux (like Debian GNU/kfreeBSD).

Appendix A: Older headers are totally fine

Fun fact: the packages `linux-libc-dev` or `kernel-headers` mentioned above (those that contain the files that `make headers_install` installs) sometimes look outdated in Linux distributions, as they sometimes stay on a older version while way newer kernels with higher version numbers get installed while updating the system. Don't worry about it, as there is a reason for this. It's actually explained in the kernel documentation file referred to above:

```Kernel headers are backwards compatible, but not forwards compatible. This means that a program built against a C library using older kernel headers should run on a newer kernel (although it may not have access to new features), but a program built against newer kernel headers may not work on an older kernel.```.

In other words: don't update `linux-libc-dev` or `kernel-headers` if you want to ensure code you compile works on older Linux kernel versions. That for example might be the kernel found on and installed by the installation ISO, which people will use before they update their systems (which some users never do). But don't worry too much about this aspect, for most software that won't matter anyway.

Appendix B: Why did nobody upstream the concept behind `linux-headers`/`kernel-devel`?

Still reading? Seems like you are really curios! Good, because I got something more for you.

I explained `make headers_install` above and mentioned the Linux kernel has no make target to install the files that are shipped in packages like `linux-headers` and `kernel-devel`. I don't known why such a make target is missing upstream, but I suspect it's the usual: nobody set down to clean one approach up and submit it upstream (apart from that approach used by the `make bindeb-pkg` target).

But it's IMHO clearly something that would be useful to have upstream: all big distro have packages like that and create it manually, which is error prone; having such a make target upstream would also be handy for people that compile and install their kernel locally, too.

I more than once considered upstreaming this functionally, but never found the time to do so. Or, to be more precise: I never found the motivation(™). ;-)

For the foreseeable future that won't change. I hope someone else will pick this task up. How about you? I guess it would be a good learning exercise and shouldn't be too hard. Looks to me like you might not even have to know C to implement this! Knowing a bit or two about Makefiles might come in handy, but I guess even that is not needed for someone really motivated!

Closing words: If you spot anything here that you consider false or misleading please let me know; especially when it is in one of those sections about the Debian-universe and its packaging, because that's not really my area of expertise. I'm happy to update this text to fix mistakes or provide additional details if they are important to know in this context.

kcbench, the Linux kernel compile benchmark, version 0.9.0 is out

Hello, is this thing still on? Looks like I have not blogged here in nearly 10 years. Uhhps. But today there is a reason to write something again:

I released kcbench 0.9.0. Kcbench is a simple Linux kernel compile benchmark that downloads the Linux sources and measures the time it takes to build the kernel. This is how it looks:

One can use the benchmark to compare different machines, just make sure they use a similar compiler. Kcbench can perform stress tests, too. It's also useful to find the optimal number of jobs for compiling source code, as 'just use all cores' sometimes is not the fastest setting.

For example, on a AMD Ryzen Threadripper 3990X (64 cores/128 threads) a lot of people will expect compiling with 128 jobs (`make -j 128`) will be fastest. But while testing kcbench on a such a processor it turned out that at least the testes machine built a Linux kernel quite a bit faster with only 64 jobs (and thus utilizing only the real cores):

There was no time for a closer inspection, but it looks like memory and its caches might have been a bottleneck that lead to this. It was different on a AMD EPYC 7742 (128 cores/256 threads), as using 256 jobs there lead to the best result:

For more about kcbench see the project page at gitlab, the README and the man-pages for kcbench and kcbenchrate. The rate variant is one of the big new enhancements in the new version. Instead of compiling one kernel really fast kcbenchrate compiles one kernel on each CPU to measure the rate and keep the all cores busy all the time. See the man-pages for a more detailed explanation of the approaches the two use. Cross-compilation is supported now, too. To learn about all the other improvements in kcbench v0.9.0 see its release page.

The new kcbench including kcbenchrate is in Fedora Rawhide. It's also in updates-testing for Fedora 31 and Fedora 32 currently and will be moved to the proper updates repositories in a few days.

Why I'm not posting much on Google+ – or – My big problem with Google+

The "tldr"-version: Google+ has lots of nice things, but in its current state it does not look like a solution that could replace twitter (and identi.ca) for me at all. In fact I'd need something like tags or topic-streams (that people could select when adding me to their circles) for my messages before I'd feel more comfortable to write public posts on G+ more that occasional.

The detailed version: It seems the people that added me to their circles mainly know me from one of four aspects of my life:

1. my work for heise.de and c't
2. English translations of that work published on "The H"
3. my contributions to Fedora
4. this strange thing called real-life (good 3D effects!)

Sending out just one public information stream to those people afaics would only make very few of them happy; I'd feel like spamming them with crap they are clearly not interested in.

Among the reasons for that view are simple language problems: I was born and live in Germany and speak German most of the time – but I know lots of people from the US, the UK and other countries and most of them do not know more that five German words. Many of them added me to their circles nevertheless; I'm also in a lot of circles where I don't know the people at all; hence I don't know their interest in me or their language preferences. I could avoid annoying them with German posts by writing in English all the time when doing public posts – but that's a crappy workaround (one I suppose might annoy some of my German friends over time) and not a real solution for the underlying problem.

Yes, you are of course right, that is a problem on Twitter as well. But I solved that easily by using multiple accounts right from the start, which is not easy with G+. Using three accounts also mostly circumvents the problem I indicated in the beginning: Different people are interested in different things and only they can decide what they are interested in.

Here is how I do it with twitter:

• My main account is @thleemhuis. Just like everybody else I use it to tweet things from my real life (including work). Most of the tweets are German, some are English; almost all of the followers are from Germany or understand German afaics
• I use @kernellogauthor to tweet about things in the Linux kernel area I stumble upon; it is a kind of bonus and an additional communication channel for readers of my Kernel-Log (DE, EN). Almost all tweets are English (I assume most Germans that read the Kernel Log know English well enough) and kernel related. It has 266 subscribers (some more on identi.ca) and only a handful of them follow @thleemhuis as well
• Many English speaking people also know me from all the contributions I did to Fedora in the past few years (I'm mostly inactive in that area right now); I follow them via @knurd666 and provide them with a English tweet there now and then; only a handful of them follow @thleemhuis or @kernellogauthor
  

IOW: I have three kinds of followers (something like 500 on twitter in total and 200 more on identi.ca) that seems to be interested in stuff I do, but there are only very few people (I assume less than 5) that follow all three accounts; and yes, I occasionally mention the different accounts on each other, so it's not a secret I use multiple accounts...

I'd really like to feed them with information from my life or work via Google+, too, but I can not see how to do that efficiently, because right now I have to know that people are interested in; on twitter those that follow me can decide on their own. That's why G+ seems so alien and wrong to me. Something like predefined tags, streams or "outgoing circles" people could select when they add me could help. But maybe that's to complicated; and maybe my situation is special and shows a problem other do not encounter.

P.S.: Yes, maybe I'm thinking to much as someone that wants to maximize the group of people he can reach -- maybe that has something to do with my day ob ;-)

New Intel graphics drivers released

The Fedora developers among you will be aware: there is a Fedora Test Day with a focus on the Intel graphics card driver today (Thursday the 30th). The X.org driver that is being tested afaics is xorg-x11-drv-intel (also known as xf86-video-intel) version 2.12.0, release 6.fc14 -- the latest F14 build as of today.

Oh, and something else happened today: After three months of development Carl Worth released a xf86-video-intel version 2.13.0. Quoting the release announcement: "[...] With the many bug fixes in this release, we encourage everyone using 2.12 to upgrade to 2.13. [...]"

So people that participate in the testing today might run into bugs that that are fixed in 2.13 or in its pre-release already. But what is even worse from my point of view: With the new and recently ratified Fedora updates policy it seems unlikely to me that Fedora 14 will update to 2.13 ever (note: unlikely still means it's possible!).

My head simply fails to understand why that is the right thing to do in this and similar cases. Even worse: lot's of other distributions have similar policies, so it's take months or years till the fixes the Intel developers worked out in the past three months make their way to regular end users.

Visiting LinuxTag and Red Hat Summit 2010

Blog entry for my fellow readers that visit LinuxTag (next week) or Red Hat Summit/JBoss World 2010 (starting in two and a half weeks from now):

• I'm in Berlin for LinuxTag next Saturday (on my own efforts).
• I'm in Boston for the whole Summit to report about it for my employer.

So if you want to meet me in real life simply look out for me or sent a mail to arrange a meeting.

BTW, looks like I might stay in Boston till Sunday afternoon, so I will have some free time once the Summit is over and all work done. Please let me know if you have any good suggestion how to best spent that time it in Boston! tia!

What questions would you like to ask the Candidates for the Fedora Board, FESCo, and FAMSCO?

As you may have heard already, several seats of the Fedora Board, FESCo, and FAMSCO are up for election soon(¹). Right now we are in the nomination period, which will be followed by a "Candidate Questionnaire." That means we'll give candidates a list of questions to answer by private mail within one week after the nomination period closed; the results will be publish soon after that to make sure they are available to the public before the Town Hall meetings on IRC happen.

Candidates may choose to answer (or not) those questions as they see fit. Voters can use the answers to get an impression of what the candidate think or plan to do while serving for the committees they are nominated for. That should help to get a interesting discussion running during the IRC Town Hall meetings; furthermore, those people that can't or don't want to participate in the IRC meetings can use the answers to make a more informed vote.

Hence we need to prepare a few good questions that we can send to the candidates once the nomination period ends. And that's where I need *your help* now:

If you have one or more questions you'd like to send to the candidates simply go and add them to:

https://fedoraproject.org/wiki/Elections/F13_Questionnaire

It just takes a minute or two, so best to do it right now -- otherwise you might get distracted and forget about it. ;-)

I'll take care of the remaining work to review, sort, and clean up the questions(²); after that I'll send them to the candidates soon after the nomination period ended. Hence, I need your question suggestions by around the 15th November 17:00 UTC latest to get a chance to prepare everything in time.

So please go to the wiki now and add at least one hard question! The answers will help Fedora contributors to chose whom to vote for! Thanks in advance for your help .

CU
knurd

(¹) If you haven't read about it yet see
https://fedoraproject.org/wiki/Elections for details.

(²) If you want to get involved or review the questions before I send them please drop me a line and I'll try to get that arranged; maybe we can arrange a quick, informal IRC meeting on Sunday evening if there is interest

P.S.: This blog post is mainly meant to get spread via planet.fedoraproject.org ;-)

New RPM Fusion packages: nvidia drivers for F12, staging drivers for F11 and F12

Just FYI:

• the nvidia graphics drivers finally showed up again in the RPM Fusion repository for rawhide (the current public rawhide to be precise, e.g. what becomes Fedora 12 soon)
• Most drivers from the linux-staging tree are disabled in the kernels that Fedora ships (among them a few wifi drivers like rtl8187se) . That's a good thing, as they are often of highly questionable quality (¹, ²). But some people nevertheless what them, as they own hardware that needs them(³). Those people from now on can get them easily for Fedora 11 and Rawhide/Fedora 12 by installing kmod-staging from RPM Fusion.
  Please note that:
  
  
  
  • the drivers for Fedora 11 are in rpmfusion-free-updates-testing currently and build for the kernel that is in Fedora's updates-testing repo; thus to use them you need to run something like "yum --enablerepo='*testing' install kmod-staging" and reboot into the kernel that is installed to use them
    
  • you need to install kmod-staging-PAE if you use a PAE kernel on your x86-32 machine
    
  • there are no akmod-staging package at this time
    
  • the package doesn't contain all the drivers from the staging tree; in case you miss one just file a bug in bugzilla.rpmfusion.org and tell the packager to enable it
    

Also note that it might take some hours till the mirror yum chooses for you offers the packages, as they were uploaded to the master repo just a few minutes before this blog posting got published.

We thank you for your attention and we wish you a pleasant flight.

(¹) that's the long story short

(²) don't expect the drivers to work well; NetworkManager for example will in have problems with some of the WiFi drivers in staging-kmod. In most cases that will be the fault of the driver and not NetworkManager, thus filing NetworkManager bugs that occur with staging drivers is likely a waste of your time.

(³) friends don't let friends buy hardware which need's staging (or even worse: proprietary) drivers on Linux