Package names can be false friends, too, as (linux|kernel)-headers show

TLDR: The `kernel-headers` packages in Fedora, RHEL, or CentOS do not carry the files you get by installing `linux-headers-$(uname -r)` on Debian based distros. If you want to build add-on modules for the kernels in Fedora, RHEL, or CentOS, you need to install `kernel-devel` instead. Fun fact: I consider both approaches to package naming flawed.

False friends

Packages in different, unrelated distros sometimes have similar names, but  nevertheless contain something entirely different. People that don't keep this in mind are sometimes going to have a bad time. 

There is one area where this afaics causes trouble quite often: when building add-on modules for your distro's kernel. On Debian or Ubuntu quite a few people learned: before starting the compiler, one has to install additional files by running a command like `apt-get install linux-headers-$(uname -r)`. That's why it's easy to assume the `kernel-headers` package fulfils the same job in Fedora, RHEL, or CentOS. But it's a wrong assumption and will lead to failure, because `kernel-headers` packages in the RH-universe contains something totally different.

The two package names are thus something like 'false friends': they look or sound similar, but differ significantly in meaning.

You actually need to install a package called `kernel-devel` to build add-on modules for the kernel from Fedora, RHEL, or CentOS . They also ship a `kernel-headers` package, but don't bother with it, it's not directly needed.

Neither package name is fully accurate

You now might wonder, which of the two approaches in naming is corrent: `linux-headers-foo` or `kernel-devel`? The short answer from my point of view: neither, as both names are a bit off.

To understand why, let's take a closer look at the Linux kernel – that what is being packaged here and thus should be a strong indicator of what's right or wrong. When looking at the sources you'll actually find something that is relevant in this context: a make target called `headers_install`. The help text describes it as 'Install sanitised kernel headers to INSTALL_HDR_PATH'.

That doesn't tell much, so people familiar with `linux-headers` packages might quickly jump to conclusions like 'that will install the files needed for building add-on modules'. But that not the case, as the files installed by this target are something entirely different: the ```kernel headers for use by userspace```. They ```[…] describe the API for user space programs attempting to use kernel services. These kernel header files are used by the system’s C library (such as glibc or uClibc) to define available system calls, as well as constants and structures to be used with these system calls. […]```.

Those quotes are from the Linux kernel documentation. To read them in full context head over to `Documentation/kbuild/headers_install.rst` in the Linux sources or to its rendered version on kernel.org.

The files installed by `make headers_install` are not really relevant when it comes to building kernel modules, but crucial in a totally different situation: when building your system's C library. That's why 'Linux from Scratch (LFS)' installs them right before compiling the GNU C Library (glibc).

The Debian-universe ships the files installed by `make headers_install` in a package called `linux-libc-dev`. The RH-universe on the other hand ships them in a package called `kernel-headers`.

Missing link

So if `make headers_install` does not install the files needed for building add-on modules, what make target will install them? Thing is: the Linux kernel does not offer a make target to do just that. It never has.

The concept of shipping all the files for building add-on modules originated in the distro world somewhere. I don't known when and where exactly. I first saw it in Fedora in its early days (2003? 2004?), when Arjan van de Veen was maintaining the Fedora kernel (IIRC). He simply made the RPM install section collect the files needed for building modules and shipped them in a new sub-package of the kernel package. The latter was and still is shipped in a package called `kernel` and the sub-package got the name `kernel-devel`. That's why you need to install this package when you want to build add-on modules.

That approach followed a familiar pattern in the RH-universe, where development files are always in '-devel' sub-packages. It's thus totally normal for users to install `foo-devel` if they want to build something for or on top of a software shipped in a package named `foo`. It's not that different from the Debian-universe, which uses '-dev' instead.

Choice of names

Other distros also started packaging the files to build add-on kernel modules, but chose other names for the sub-package. The Debian-universe, which ships the Linux kernel in a package called `linux`, settled on packages with the prefix `linux-headers`. I don't known how it came to this or if `linux-dev` was even considered.

Is 'headers' an appropriate suffix, even if `make headers_install` installs something different? 

Decide yourself. In Fedora 33 the kernel-devel package currently contains about 16.500 files. About 12.000 of them indeed are header files (their file names end with '.h'). Then there are a bit more than 4100 files called `Makefile` or `Kconfig`, which are needed for configuration and building. Then there are about 80 files containing C code and about 250 other files: some scripts (perl, python, coccinelle), more Kbuild stuff and a few other things. 

In other words: there is a bit more than headers in there, but they dominate. Nevertheless I still think it wasn't the best idea to use `linux-headers` as prefix. That's mainly for two reasons: (1) a 'linux-dev' prefix would have been more in line with the approach used by other packages in the Debian-universe; (2) it's confusing users, as the kernel installs something different when running `make headers_install`. 

But to be fair on the second point here: I did not dig down into the history books, but I strongly suspect the prefix `linux-headers` was chosen and established before `make headers_install` was introduced; that was in September 2006 with Linux 2.6.18.

But in the end it doesn't matter if I find the term appropriate or not: in the Debian-universe the term `linux-headers` and the purpose of these packages is well established these days. Changing it is likely not worth the trouble. People just need to be made aware they need to install `kernel-devel` on Fedora, RHEL, and CentOS when they would install `linux-headers-$(uname -r)` on Debian, Ubuntu or distros based on them.

xkcd/Randall Munroe, CC BY-NC 2.5

Bonus feature: Fedora is wrong, too

Earlier I mentioned that I consider both `linux-headers` and `kernel-devel` flawed, but only explained what's wrong with the former. Well, it's a bit nitpicking, but the problem with the package name used in the RH-universe is on the other side of the dash. Nobody in the RH-universe would put Firefox in a package `browser` or LibreOffice in a package called `officesuite`. But somehow a kernel named 'Linux' was put in a package called `kernel`.

But whatever, fixing this would likely also create more trouble then it's worth. And it's only going to become a problem if Fedora ever decides to ship other kernels (like GNU Hurd or the FreeBSD kernel) in parallel to Linux (like Debian GNU/kfreeBSD).

Appendix A: Older headers are totally fine

Fun fact: the packages `linux-libc-dev` or `kernel-headers` mentioned above (those that contain the files that `make headers_install` installs) sometimes look outdated in Linux distributions, as they sometimes stay on a older version while way newer kernels with higher version numbers get installed while updating the system. Don't worry about it, as there is a reason for this. It's actually explained in the kernel documentation file referred to above:

```Kernel headers are backwards compatible, but not forwards compatible. This means that a program built against a C library using older kernel headers should run on a newer kernel (although it may not have access to new features), but a program built against newer kernel headers may not work on an older kernel.```.

In other words: don't update `linux-libc-dev` or `kernel-headers` if you want to ensure code you compile works on older Linux kernel versions. That for example might be the kernel found on and installed by the installation ISO, which people will use before they update their systems (which some users never do). But don't worry too much about this aspect, for most software that won't matter anyway.

Appendix B: Why did nobody upstream the concept behind `linux-headers`/`kernel-devel`?

Still reading? Seems like you are really curios! Good, because I got something more for you.

I explained `make headers_install` above and mentioned the Linux kernel has no make target to install the files that are shipped in packages like `linux-headers` and `kernel-devel`. I don't known why such a make target is missing upstream, but I suspect it's the usual: nobody set down to clean one approach up and submit it upstream (apart from that approach used by the `make bindeb-pkg` target).

But it's IMHO clearly something that would be useful to have upstream: all big distro have packages like that and create it manually, which is error prone; having such a make target upstream would also be handy for people that compile and install their kernel locally, too.

I more than once considered upstreaming this functionally, but never found the time to do so. Or, to be more precise: I never found the motivation(™). ;-)

For the foreseeable future that won't change. I hope someone else will pick this task up. How about you? I guess it would be a good learning exercise and shouldn't be too hard. Looks to me like you might not even have to know C to implement this! Knowing a bit or two about Makefiles might come in handy, but I guess even that is not needed for someone really motivated!

Closing words: If you spot anything here that you consider false or misleading please let me know; especially when it is in one of those sections about the Debian-universe and its packaging, because that's not really my area of expertise. I'm happy to update this text to fix mistakes or provide additional details if they are important to know in this context.

kcbench, the Linux kernel compile benchmark, version 0.9.0 is out

Hello, is this thing still on? Looks like I have not blogged here in nearly 10 years. Uhhps. But today there is a reason to write something again:

I released kcbench 0.9.0. Kcbench is a simple Linux kernel compile benchmark that downloads the Linux sources and measures the time it takes to build the kernel. This is how it looks:

One can use the benchmark to compare different machines, just make sure they use a similar compiler. Kcbench can perform stress tests, too. It's also useful to find the optimal number of jobs for compiling source code, as 'just use all cores' sometimes is not the fastest setting.

For example, on a AMD Ryzen Threadripper 3990X (64 cores/128 threads) a lot of people will expect compiling with 128 jobs (`make -j 128`) will be fastest. But while testing kcbench on a such a processor it turned out that at least the testes machine built a Linux kernel quite a bit faster with only 64 jobs (and thus utilizing only the real cores):

There was no time for a closer inspection, but it looks like memory and its caches might have been a bottleneck that lead to this. It was different on a AMD EPYC 7742 (128 cores/256 threads), as using 256 jobs there lead to the best result:

For more about kcbench see the project page at gitlab, the README and the man-pages for kcbench and kcbenchrate. The rate variant is one of the big new enhancements in the new version. Instead of compiling one kernel really fast kcbenchrate compiles one kernel on each CPU to measure the rate and keep the all cores busy all the time. See the man-pages for a more detailed explanation of the approaches the two use. Cross-compilation is supported now, too. To learn about all the other improvements in kcbench v0.9.0 see its release page.

The new kcbench including kcbenchrate is in Fedora Rawhide. It's also in updates-testing for Fedora 31 and Fedora 32 currently and will be moved to the proper updates repositories in a few days.

Why I'm not posting much on Google+ – or – My big problem with Google+

The "tldr"-version: Google+ has lots of nice things, but in its current state it does not look like a solution that could replace twitter (and identi.ca) for me at all. In fact I'd need something like tags or topic-streams (that people could select when adding me to their circles) for my messages before I'd feel more comfortable to write public posts on G+ more that occasional.

The detailed version: It seems the people that added me to their circles mainly know me from one of four aspects of my life:

1. my work for heise.de and c't
2. English translations of that work published on "The H"
3. my contributions to Fedora
4. this strange thing called real-life (good 3D effects!)

Sending out just one public information stream to those people afaics would only make very few of them happy; I'd feel like spamming them with crap they are clearly not interested in.

Among the reasons for that view are simple language problems: I was born and live in Germany and speak German most of the time – but I know lots of people from the US, the UK and other countries and most of them do not know more that five German words. Many of them added me to their circles nevertheless; I'm also in a lot of circles where I don't know the people at all; hence I don't know their interest in me or their language preferences. I could avoid annoying them with German posts by writing in English all the time when doing public posts – but that's a crappy workaround (one I suppose might annoy some of my German friends over time) and not a real solution for the underlying problem.

Yes, you are of course right, that is a problem on Twitter as well. But I solved that easily by using multiple accounts right from the start, which is not easy with G+. Using three accounts also mostly circumvents the problem I indicated in the beginning: Different people are interested in different things and only they can decide what they are interested in.

Here is how I do it with twitter:

• My main account is @thleemhuis. Just like everybody else I use it to tweet things from my real life (including work). Most of the tweets are German, some are English; almost all of the followers are from Germany or understand German afaics
• I use @kernellogauthor to tweet about things in the Linux kernel area I stumble upon; it is a kind of bonus and an additional communication channel for readers of my Kernel-Log (DE, EN). Almost all tweets are English (I assume most Germans that read the Kernel Log know English well enough) and kernel related. It has 266 subscribers (some more on identi.ca) and only a handful of them follow @thleemhuis as well
• Many English speaking people also know me from all the contributions I did to Fedora in the past few years (I'm mostly inactive in that area right now); I follow them via @knurd666 and provide them with a English tweet there now and then; only a handful of them follow @thleemhuis or @kernellogauthor
  

IOW: I have three kinds of followers (something like 500 on twitter in total and 200 more on identi.ca) that seems to be interested in stuff I do, but there are only very few people (I assume less than 5) that follow all three accounts; and yes, I occasionally mention the different accounts on each other, so it's not a secret I use multiple accounts...

I'd really like to feed them with information from my life or work via Google+, too, but I can not see how to do that efficiently, because right now I have to know that people are interested in; on twitter those that follow me can decide on their own. That's why G+ seems so alien and wrong to me. Something like predefined tags, streams or "outgoing circles" people could select when they add me could help. But maybe that's to complicated; and maybe my situation is special and shows a problem other do not encounter.

P.S.: Yes, maybe I'm thinking to much as someone that wants to maximize the group of people he can reach -- maybe that has something to do with my day ob ;-)

New Intel graphics drivers released

The Fedora developers among you will be aware: there is a Fedora Test Day with a focus on the Intel graphics card driver today (Thursday the 30th). The X.org driver that is being tested afaics is xorg-x11-drv-intel (also known as xf86-video-intel) version 2.12.0, release 6.fc14 -- the latest F14 build as of today.

Oh, and something else happened today: After three months of development Carl Worth released a xf86-video-intel version 2.13.0. Quoting the release announcement: "[...] With the many bug fixes in this release, we encourage everyone using 2.12 to upgrade to 2.13. [...]"

So people that participate in the testing today might run into bugs that that are fixed in 2.13 or in its pre-release already. But what is even worse from my point of view: With the new and recently ratified Fedora updates policy it seems unlikely to me that Fedora 14 will update to 2.13 ever (note: unlikely still means it's possible!).

My head simply fails to understand why that is the right thing to do in this and similar cases. Even worse: lot's of other distributions have similar policies, so it's take months or years till the fixes the Intel developers worked out in the past three months make their way to regular end users.

Presentations from LinuxTag / Fudcon Berlin 2009 available; Tomorrow: Hackfests

My presentations on LinuxTag and Fudcon Berlin 2009 went mostly well. Biggest problem when looking back now: Seems it was more then enough stuff to talk for 90 or 100 minutes, but I only had a hour in total :-/.

Want to look at the slides? Follow these links:

• German, LinuxTag: "Überblick über aktuelle Entwicklungen des Linux-Kernels"
• English, FUDCon: "What is RPM Fusion and why it is really important for Fedora (and how you can help to make it better!)"

Another problem: There are so many people here at FUDCon and I more and more get the impression that the list with people I "want to meet, shake hands with and talk to for a while, to get a better connection between faces, email addresses, nicks, and (those sometimes overrated) real names" isn't getting shorter as fast as it should to meet everyone in the remaining time. Seems I need to speed up somehow...

It's the same for you? If yes and if you ever wanted to talk to the crazy troublemaker that did or does a lot of work for Fedora, EPEL and Livna/RPM Fusion then just grab me and start to talk to me when you see me walking by or sitting somewhere. The photo on the right shows how I look like, just imagine some mostly inconspicuous glasses on the nose. Ohh, and the hair is a bit shorter right now (I guess nature will do it's best to fix that over time).

BTW, again some informations for the FUDConners: I plan a small "RPM Fusion" hackfest for tomorrow *if* people are interested in one. There is no real plan what to do exactly besides the general idea: Do some work to improve RPM Fusion, and work out the details on the fly. Want to join? Then just watch the wiki and the usual places at FUDCon for details!

thl and knurd @ LinuxTag 2009 and Fudcon Berlin 2009

I just arrived in Berlin at the fairgrounds where LinuxTag 2009 and Fudcon 2009 are held. I'm staying till Sunday evening, but like so often I have to split my attention and time as I'm partly here for my employer and partly for fun/Fedora (just like last year at Red Hat Summit Boston).

As usual that's creating some interesting problems where I ideally wound like to be at two places at the same time :-/

Friday will get most complicated. I for example can't visit the Fudcon-Keynote from Jan Wildeboer, as I'm giving a "Kernel-Log" talk as part of the "Kernel Track" of LinuxTag at just the same time. That wasn't planed like that, but seems Jonathan Corbet canceled his trip and thus can't give his well know and always interesting "The Kernel report"; I three or four weeks ago was asked to fill the position and accepted (presentation is basically finished; luckily Linus closed the merge window a few hours ago). Note that I'm giving that talk in German, not in English, as Jonathan would have done.

I also have time and space problems for the barcamp pitching session at 5 pm local time: Together with Nils Magnus from LinuxTag e.V. / Linux New Media I'm moderating the Kernel Kwestioning at that time, where the audience can ask some kernel developers on stage a few questions.

And that are only the hard scheduled events which I really can't miss. I also want (have?) to hear at least two of the other talks (Union Mounts and Ksplice) from the Kernel track on LinuxTag, so I likely walk back and forth between Kernel track and Fudcon a lot on Friday. But I hope to join the Fudcon crowd early for Fudpub in the evening -- I mean, I likely *have to*, as the Fudpub otherwise might run out of pizza and drinks before I arrive (you know how Fedorians are...).

Remains to be seen how long I can stay at Fudpub, as I still have to prepare my presentation on RPM Fusion which I'm giving at 2 pm on Saturday on Fudcon. I'll give that talk in English -- I'm not completely sure yet if that works out, but I guess I have to give it a try sooner or later. So remember to bring eggs and tomatoes to throw them on the stage if my English is way to worse ;-)

So, see you on LinuxTag and Fudcon! This is how I look like. But note, I had to buy a add-on a few months ago: now with glasses (often, not always) ;-)

P.S.: Headline-Explanation: those that contribute to Fedora for a long while will remember that I used to be known as "thl" in the Fedora -world initially. But I already was and still am "thl" at work and a few years ago decided it might be better to use a different nick in the Fedora world -- after a bit of thinking I chose "knurd". That way it's at least a bit harder (especially on IRC!) to get the connection "Ohh, the thl that writes the kernel-log for c't/heise.de/the-h is also the one that contributes to Fedora", which I think is a good thing. But it's not really a secret anyway, as rare blog entry like this make it obvious. But trying to keep it a real secret would not work anyway, as the name "Thorsten Leemhuis" quite rare, so it's not that hard to find out anyway.

rpmfusion-{,non}free-remix-kickstarts -- or -- knurdix

There was always the idea to build a linux distribution with RPM Fusion packages within the RPM Fusion project -- e.g. live media and installer spins ^w remixes that are basically a distribution like Fedora under a different name and with some packages from RPM Fusion. There is/was Omega 10, which until now never became the official spin for one reason or another. Recently some other work was done to get one step closer to create a official "Fedora Remix" from RPM Fusion sooner or later:

RPM Fusion since a few weeks contains the packages rpmfusion-free-remix-kickstarts and rpmfusion-nonfree-remix-kickstarts in the repositories for Fedora 11 and Rawhide. They just like the Fedora package spin-kickstarts contain kickstart files that can be used to create your own linux distribution live-image using the Fedora Package Collection and Fedora's livecd-tools. A installer image is still on the todo list and might need some fixes in anaconda and the rpmfusion-{,non}free-release packages afaics.

The kickstart files in the two RPM Fusion packages are pretty basic: They simply include the kickstart files from Fedora, remove the Fedora branding, add the generic branding and add the repo definitions for RPM Fusion. That's basically it, because the comps groups from RPM Fusion extend the groups that Fedora defines and thus you automatically get all the packages you want depending on what's defined in the RPM Fusion comps files -- that is for example gstreamer-plugins-ugly and gstreamer-ffmpeg in the case of the Gnome groups and xine-lib-extras-freeworld for the KDE remix.

As a proffce of concept I created four remixes (Desktop and KDE for i586 and x86_64) for testing purposes and uploaded them to to the web for public testing. But warning: They are basically untested and just meant to show what's possible. Ohh, and due to the lack of a better name I just called them "knurdix" ;-) We'll need to find a different name for the official RPM Fusion images (which might be simply "RPM Fusion" or something else that sounds better).

Hopefully those packages and the kickstart files in it can help to get some people interested in the idea of a RPM Fusion remix -- maybe enough people to build and maintain official "Fedora remixes" with packages from RPM Fusion" in the long term. If you are interested to help just join the rpmfusion-developers list and share your ideas. Or get in contact with me directly, but the mailing list is the preferred way.

Leave kmods in RPM Fusion, but make sure they work well

In short: Fedora IMHO should not ship separate packages with kernel modules (so called "kmod packages") as part of the package collection. But Fedora afaics needs to make sure things like kmods work well, even if they are not used in the Fedora package collection.

Verbose variant: Recently there was another(¹) discussion on fedora-devel about allowing packages with kernel-modules in the Fedora package collection. I'm one of the drivers of the original "kmod" packaging standard for kernel-modules in Fedora and take care of it in RPM Fusion these days (which uses a modified version). So obviously I have a strong interest in the area.

Nevertheless I stayed away from the discussion and I guess many would have been surprised if I'd shared my option: It doesn't make much (if any) sense to package kernel modules as separate packages in Fedora. The main reason for this option: Everything lands in one package repository anyway(²), so it's much easier for everyone to get the bits into Fedora's proper kernel package directly. And I agree with the suggested approach way to make that happen as well: get the modules upstream, e.g. in linus' "vanilla" kernel (and not the staging area), then Fedora will get them automatically.

But: Some widely used kernel-modules will never make it upstream. For others (like some of those in linux-staging) it's a long way that takes a lot of time, because the code of those modules often is in a bad shape; until that's fixed and an improved variant merged a lot of people are nevertheless willing to use the ugly code we have today, because for them it's often better than throwing hardware away.

Fedora ignores those cases -- afaics because they are "not the right thing to do" or maybe "politically incorrect". One can argue if that is right or wrong for Fedora, but that's a topic for a different blog entry. The real point I'm up to: users still want to get the modules they need. Thus they will either work around it locally, use/contribute to repositories like RPM Fusion, or switch to a distribution that offers them what they want.

The latter is something that is neither in the interest of Fedora or Red Hat. Hence I'd say it would be good if Fedora would make sure that things like kmods work well, *even if they are not used within Fedora* -- just like Red Hat makes sure that Oracle DBs work fine on RHEL (which uses kmods in RHEL5, so RHEL would benefit from proper kmod support and testing in Fedora as well). But that's not the case right now(³), which makes Fedora sometimes annoying or hard to use.

(¹) There have been a lot similar discussion around kmods over the past years.

(²) My option might be different if we had something like a "extras"/"unsupported" add-on repositories within Fedora, as that would make it obvious "hey, these modules are not part of the upstream kernel/might not work so well/might suddenly go away again".

(³) Note that the main problem afaics is on the Fedora side, so it's not really fixable for RPM Fusion. I outlined the details in a posting to fedora-devel last August, but nothing happened. There are ways to work around the problem using a yum-plugin. But that is not a real fix, because the problem is not specific to kmods -- it happens with a lot of other packages as well in cases where a RPM Fusion package depends on a specific version of a package that is part of Fedora (xine-lib and xine-lib-extras-freeworld for example). And yes, sorry, I don't have the skills to fix the problem myself. But I don't think I have to -- Fedora afaics is meant to be a community where people with different skills work on different areas of the project to make the whole thing better.

Fedora 11, kernel-PAE, and what it means for your x86-32 system

There is one small change in Fedora 11 that I guess will confuse Fedora and RPM Fusion users with x86-32 (aka i386/ix86) systems quite a lot, but afaics did not get enough attention yet:

• By default, the PAE kernel will be used on 32-bit hardware, where appropriate(¹).

"Appropriate" is not really explained -- maybe because it's a bit hard to sum up without going into the boring details. But basically it boils down to: The kernel with PAE support will be installed by Fedora 11 for x86-32 on the majority of x86-compatible systems that have been manufactured in the past three or four years(¹). So likely on your system as well if you are running a x86-32 distro on a modern system.

The important part: the package containing the kernel with PAE support is not called "kernel" -- it's called "kernel-PAE" instead. And that's not the only package where "-PAE" is used as suffix. That has major certain consequences on systems where Fedora 11 installs kernel-PAE:

• when you build kernel modules with akmods, dkms, or manually, then you from now on need to install kernel-PAE-devel instead of kernel-devel
• similar for kernel-modules: instead of "yum install kmod-nvidia" your now need to type "yum install kmod-nvidia-PAE". Yum otherwise will try to install the matching kernel without PAE support for you, which (in short without the boring details) is something you most of the time don't want(²).

In other words: the change in Fedora 11 makes lots of howtos, FAQs, articles on the net and in computer magazines confusing, wrong, misleading or harmful (depending on view and specific howto/FAQ/article), because most of those docs don't consider the above fact (yet).

And that's not Fedora's fault -- PAE kernels are around for a long time in Fedora already. But they were used only on a minority of systems. Most (not all!) of those that have written today's howtos, FAQs or articles were likely either not aware of it or chose to ignored it to keep things simple.

That's backfiring now. So go and spread the news on mailing lists, forums and other places where it might be of interest. Feel free to copy-n-paste this whole text or simply point to this blog entry. Thanks in advance!

(¹) E.g. since processors with NX bit became mainstream; NX stands for NoeXecute and is also called Enhanced Virus Protection by AMD and xD-Bit by Intel

(²) not sure, but maybe the yum-plugin "yum-fedorakmod" could have made yum to the right thing and install the proper kmod for the PAE kernel. I never tried and it doesn't matter much as the plugin is not available in the Fedora or RPM Fusion repositories for F11. If someone wants take care of yum-fedorakmod and wants to get it into RPM Fusion then please drop me a line.

Unprofessional

Dear GNOME project,

thanks for running the "GNOME 3.0 General Sociological Research" survey and putting the results online.

A small note: If you ask people if they use foo, bar, baz or foobar and most people select foo, that that's it. Nothing more. And definitely don't conclude "Nearly ³/₄ of the people use foo for it's stability, compatibility and ease of use", because the latter part is a personal option if there wasn't a question "why do you use foo" with checkboxes "stability", "compatibility" and "ease of use" next to it.

Personal opinions like that render the whole conclusion unbelievable and make things look highly unprofessional.

rpmusrtools?

The rpmdevtools are in Fedora for quite a while now and really helpful in a lot of situations. So are the tools in the yum-utils. But the tools in those two packages don't completely satisfy all my needs, hence I (ages ago) wrote a few small scripts that do some additional magic:

• run "rpm -V" on all the packages and look out for packages where files are (a) missing or (b) corrupted; the script as a bonus (c) can even print out config files that have been modified (which likely should be part of a separate script, as that might be handy for backup purposes); if likely (d) should offer to reinstall packages that have missing or corrupted files
  
• run something like "find /bin/ /boot/ /etc/ /lib/ /opt/ /sbin/ /usr/ /var/" with "-type f" and/or "-type d" and let rpm check if the list of files and/or directories that find outputs are owned by a package -- complain if not; that's quite a lot of work for the system and takes a while to complete, but often turns up a whole lot of old and obsolete cruft that is lying in some hidden corner of the hard disk just waiting to get erased (just checked a random machine; 3241 files sum up to 380 MByte...)
  
• run something like "find /boot/ /etc/ /opt/ /usr/ /var/" with "-name '*.rpmnew'" "-name '*.rpmsave'"; go through the list of files find outputs, show a "diff -u" and offer the user to either
  
  
  • run tools like meld or kdiff3 to merge the files(¹) and remove the .rpm{new,save}-file afterwards
    
  • replace the file with its .rpm{new,save}-pendant (or do it semi-automatically if sha1sums are identical)
    
  • delete the .rpm{new,save}-file
    
  • do nothing
  
  
• a script that calls all the above tests as well as "package-cleanup" sequentially with the options "--problems", "--orphans", and "--dupes" to generate a report of "unusual" things
  

All my scripts are quite rough and not really ready for general consumption. But I wonder if it would be worth to collect scripts like these (a lot of users likely have similar scripts), pick the best ones, clean them up in a fedorahosted project and ship them in a package "rpmusrtools" within the fedora repositories.

Dear reader: Would you find such a package useful? Or are there already such scripts/tools somewhere in the big Fedora package collection and I'm just not aware of them?

(¹) that works quite well for a lot of config files, but breaks horrible for those that have lots of comments in them that try to explain each and every option and/or variable :-(( it get even worse if those comments change every few months.

So this goes to the authors of dovecot, postfix, squid, and other software that has lots of comments in their default config files: "Config files are for configuration and should not contain lots of comments that get adjusted every few months, as those make it really hard to merge older (modified) config files with new one; in the future please consider to explain all option in a proper document like a man-page instead please and remove the comments from the config files! Thanks in advance! Yours truly, Thorsten Leemhuis"

I really wish we could get rid of ".fcXY" as disttag in rawhide

I really wish we would get rid of using ".fcXY" as disttag in rawhide because it confuses way to many people when they find packages with tags like "fc7" on their fresh rawhide install... Simply using ".1" as disttag imho would be the best solution.

$ rpmdev-vercmp 1.2-3.1 1.2-3-fc10
0:1.2-3.1 is newer

Then packages that (for good reasons) don't get rebuild during one or two devel cycles don't have disttags from older fedora releases in their package names. But people didn't like it back when I proposed it some years ago. :-/ Hopefully someone sooner or later comes up with a better idea. ".fcn" for "Fedora Collections newest" maybe?

$ rpmdev-vercmp 1.2-3.fcn 1.2-3-fc10
0:1.2-3.fcn is newer

Some notes on RHEL 5.3

Congrats to the RHEL team for getting RHEL 5.3 out. Some random thoughts that sprung to my mind when looking closer at it:

• The release announcement (short), the Red Hat Enterprise Linux 5.3 Technical Overview (medium sized) and the release notes (all the details) are really great resources to see what's new in the latest RHEL release. I wish we had similar documents for each new Fedora release (yes, we have the release notes, but they contain a lot of boring details regular Fedora users already know).
  
• The kernel and dmraid in RHEL 5.3 now support dmraid 4/5/10. Alasdair, Heinz, others in RH: can you *please* speed up now to get that code (which is developed for quite some time now) upstream for Linux 2.6.30 to make sure it gets into Fedora 11 and from there into RHEL 6? I guess many users would be glad about that, as a lot of newer amd, intel and nvidia chipsets for desktop mainboards support RAID 5.
  
• Many thanks in advance to the CentOS folks that already started to work on CentOS 5.3
  
• I guess it will be an interesting comparison if somebody would compare the number of improvements (and their impact) that RHEL did between 5.2 and 5.3 with those that Ubuntu LTS 8.04.2 brings ;-) [Update 20090123]: Bad timing, 8.04.2 came out just a few hours after I wrote this; list of changes is available [/Update]
  

Communication is important

Short version: More and more decisions in Fedora are done on IRC and in other places you have to be aware of; that itself would be no problem if the mailing lists would stay in the loop to make sure people cat raise their option before something is decided. But that's often not the case. The Fedora Project needs to improve here, as proper communication between contributors and those that make the decisions is a key factor for a community project.

Long version: Cut'n'pasted below is a part from a post that some time ago was send to some random fedora mailing list by someone that's quite important in Fedora:

I get the fact that you resent being in a timezone and location that
makes it difficult for you to participate in higher bandwidth forms of
communication (irc, phone, face to face).  I can't help that.  However
I'm not about to force the entire Fedora project slow to a crawl just so
that every thought, comment, discussion, fart, whatever happens via a
public email.  That's just ridiculous.  People will continue to talk on
IRC, will continue to chat via IM, will continue to talk on phones, and
will even *shock* talk in person!  Ideas, proposals, and even a decision
or two, depending on the group, will be made in these ways.  Deal with
it.

Statements like this are one of the reasons why I'm not as active in Fedora anymore as I was two or three years ago.

Sure, the one that wrote above para made a lot of good points. Face-to-face or IRC meetings are important and often help a lot to drive things forward. And we all afaics don't want a totally bureaucracy Fedora with hundreds of rules that express how decisions or other things have to be done; I actually tend to say we actually have way to many rules in some places already and need to get rid of a few (but that's a different topic).

But in the end above para sounded to me: be on FUDCon, IRC, or right hallway at a specific time on a specific place on earth or you have no chance to influence things. It might not have been meant that way, but that's what my mind made out of when reading it. Partly that's because I got the impression that more and more things in Fedora actually work in a "be there at the right time and place if you want to get heard"-way. Obviously that's bad for a lot of contributors due to time zone differences, job, vacation, or other real life issues getting in the way; which, obviously again, is bad for a community project that has contributors from places spread all over the world, as it excludes some contributors from getting involved or from influencing decisions.

That's why mailing lists (or a similar form of time-zone-independent communication) IMHO are very important for community projects with lots of contributors. But Fedora seems to move more and more things away from the public lists to other places -- especially IRC is used more and more desperate that we know there are people that don't want or can't join IRC.

Moving things to IRC wouldn't be such a big problem if important things that come up for decision in IRC or other places get announced properly 2 or 3 days beforehand on the lists. But that's often not the case, hence people that can't make the meetings get no real chance to see what coming up -- hence they can't share their options before something is up for vote. But that's IMHO a very very important factor for a community project, as contributors want to feel "heard" and have a chance to influence stuff, as that makes them feel as part of the project (and not like a citizen of a state where you can elect the decision makers every few months or year) -- even if the outcome in the end is not what the contributor wanted initially, as everyone knows (or at least should) that you can't get everything you want every time (a pony anyone?).

What makes things even worse: Now and then there are insufficient summaries from Fedora IRC meetings; the board is the big exception here while rel-eng often is not writing a summary at all. Sure, writing those summaries is a boring task -- I know that, as I had to write a lot of them for FESCo and the EPEL Steering Committee in the old days. But it's worth it, as a quick summary (even if it misses some details) is way easier to read then a log from a IRC meeting; that helps people to know what's up which again gives them the important feeling that they are part of the project. Not even trying to write summaries IMHO is a bit like top-posting and not removing unnecessary parts when replying to a mails on a mailing list: it might be easier and quicker for you, but a lot harder and time consuming for all those out there that want to know what's up. It shows disrespect for the community.

Time and place independent communication also is not only important for IRC but also for conferences. That why I'd like to say "Thanks" to Karsten Wade aka quaid for his recent blog post "Where are your FUDCon session notes?". And also thanks to all those that put videos from the recent FUDCon sessions online. Albeit those OTOH are a bit like IRC meetings without summaries -- watching them just like reading IRC meeting logs takes a lot of time and at least for me often has a bad "benefit per time ratio" :-/ . But as I said earlier: You can't always get what you want and this is a area where that's the case ;-)

New toy

We not only got a new cat -- we (or, to be precise: my girlfriend) also got a netbook. A Samsung NC10 in white:
The Fedora 10 installed just fine -- no problems at all. Dmesg, lspci:

00:00.0 Host bridge [0600]: Intel Corporation Mobile 945GME Express Memory Controller Hub [8086:27ac] (rev 03)
 Kernel driver in use: agpgart-intel
00:02.0 VGA compatible controller [0300]: Intel Corporation Mobile 945GME Express Integrated Graphics Controller [8086:27ae] (rev 03)
00:02.1 Display controller [0380]: Intel Corporation Mobile 945GM/GMS/GME, 943/940GML Express Integrated Graphics Controller [8086:27a6] (rev 03)
00:1b.0 Audio device [0403]: Intel Corporation 82801G (ICH7 Family) High Definition Audio Controller [8086:27d8] (rev 02)
 Kernel driver in use: HDA Intel
 Kernel modules: snd-hda-intel
00:1c.0 PCI bridge [0604]: Intel Corporation 82801G (ICH7 Family) PCI Express Port 1 [8086:27d0] (rev 02)
 Kernel driver in use: pcieport-driver
00:1c.2 PCI bridge [0604]: Intel Corporation 82801G (ICH7 Family) PCI Express Port 3 [8086:27d4] (rev 02)
 Kernel driver in use: pcieport-driver
00:1d.0 USB Controller [0c03]: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #1 [8086:27c8] (rev 02)
 Kernel driver in use: uhci_hcd
00:1d.1 USB Controller [0c03]: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #2 [8086:27c9] (rev 02)
 Kernel driver in use: uhci_hcd
00:1d.2 USB Controller [0c03]: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #3 [8086:27ca] (rev 02)
 Kernel driver in use: uhci_hcd
00:1d.3 USB Controller [0c03]: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #4 [8086:27cb] (rev 02)
 Kernel driver in use: uhci_hcd
00:1d.7 USB Controller [0c03]: Intel Corporation 82801G (ICH7 Family) USB2 EHCI Controller [8086:27cc] (rev 02)
 Kernel driver in use: ehci_hcd
00:1e.0 PCI bridge [0604]: Intel Corporation 82801 Mobile PCI Bridge [8086:2448] (rev e2)
00:1f.0 ISA bridge [0601]: Intel Corporation 82801GBM (ICH7-M) LPC Interface Bridge [8086:27b9] (rev 02)
 Kernel modules: iTCO_wdt, intel-rng
00:1f.2 IDE interface [0101]: Intel Corporation 82801GBM/GHM (ICH7 Family) SATA IDE Controller [8086:27c4] (rev 02)
 Kernel driver in use: ata_piix
00:1f.3 SMBus [0c05]: Intel Corporation 82801G (ICH7 Family) SMBus Controller [8086:27da] (rev 02)
 Kernel driver in use: i801_smbus
 Kernel modules: i2c-i801
02:00.0 Ethernet controller [0200]: Atheros Communications Inc. AR242x 802.11abg Wireless PCI Express Adapter [168c:001c] (rev 01)
 Kernel driver in use: ath5k_pci
 Kernel modules: ath5k
03:00.0 Ethernet controller [0200]: Marvell Technology Group Ltd. 88E8040 PCI-E Fast Ethernet Controller [11ab:4354] (rev 13)
 Kernel driver in use: sky2
 Kernel modules: sky2

I didn't test everything yet, but suspend, hibernate and wifi worked out of the the box. Regulating the display brightness does not work using the function keys -- but it works using the gnome-applet.

Seems the wifi-LED does not work; and when I once disabled wifi using the hotkeys I could only get it to work again by restarting the system -- not sure, maybe it was just a hickup or something like that. I'll investigate further over the next few days.

Read the same paragraphs every half year?

I really wanted to read the Fedora 10 Release Notes, but when I did I quickly got distracted. Later I asked myself why and gave it a second try with the goal: watch closely why you got distracted.

I first noticed that I had missed the brief overview at the bottom of the first page. I simply had hit "Next" on the top of the first page, as I expected it to be just the index, like it's iirc is in so many multiple-page howtos -- abs is just a random example here.

I further noticed that the text it a bit hard to read, as all the links are written in plain text within the text -- so you have to skip them with the eyes when you try to read continuously. To stress this a bit more compare yourself, which to you think is quicker and easier to read:

• Features for Fedora 10 are tracked on the feature list page: http://www.fedoraproject.org/wiki/Releases/10/FeatureList

• Features for Fedora 10 are tracked on the feature list page.

I then got to section 2.1 and read: "Anaconda is the name of the Fedora installer. This section outlines issues related to Anaconda and installing Fedora 10." I don't like the bold writing -- I find is distracting. But I know, some people like that. The real problem is something else and gets even more obvious in the whole section 2.1.1: Most if not all existing Fedora users know all of that already.

And that IMHO is the big problem of the whole release notes. Sure, these information are important for new Fedora users and hence they need to be written somewhere. But these information are just boring for all the existing Fedora users out there. The new information between all the old and well know stuff on the other hand is very important to existing Fedora users -- we want them to read it (which most do not afaics). Thus I'm really wondering if we should provide a second version of the release notes that only provides information that are really new -- then users that come from the previous release only have to read that document, which like is a lot shorter.

I'm not even sure how hard it would be to create that section release note set -- maybe running a diff over the old and new version, cut'n'past the relevant paragraphs where something important changed and put them into one document.

P.S.: Another thing that I dislike in the Fedora 10 Release Notes: Why isn't there a single-page version online that would make searching something a whole lot easier. Example: I knew there was a paragraph in the release notes regarding the flash-plugin. Hence i browsed to the Fedora 10 Release Notes Index, typed "/flash" in Firefox to find it, but failed. I tried some other keywords like "plugin" and failed as well. After a while I gave up and asked Google with the search term "flash-plugin site:http://docs.fedoraproject.org/release-notes/f10/en_US/" -- then I quickly found what I was looking for. Works, but only is you know how to use Google properly :-/

Number 3: Ginger

Three and a half years ago Linus & Lucy moved in. Now they got someone to play with: Ginger




Welcome Ginger, hope you'll like it. Ohh, and Ginger, if you read this: Please stop playing on the edge of the floor above the stairs! You can fall down there easily (remember: you were close to falling down there a few times already!) and chances are big that you'll break your neck if you do.

P.S.: We considered to stick to names from Peanuts. Pig-Pen would have fit Gingers look, but well, she's (a) a girl and (b) that not really a good name for a cat. Hopefully Ginger sticks with us -- the original Ginger always tried to escape from "home".

Enable RPM Fusion during Fedora install

To all those that install the brand new Fedora 10: If you enable the RPM Fusion online repositories during install you'll in F10 and later will automatically get some of the packages installed that RPM Fusion provides. Which packages? Well, that depends on what software you choose; if you for example stick to GNOME you'll get gnome-mplayer and some gstreamer plugins by default.



For details how to use RPM Fusion during install please take a look at the documentation in the RPM Fusion wiki. It also holds some information's how to enable RPM Fusion after installing Fedora.

See also:

• RPM Fusion free and nonfree repositories for Fedora 10 (Cambridge) now available on fedora-announce-list

RPM Fusion for EL in the early testing stage

A lot of people in the past asked for a „Livna for RHEL/CentOS“. There were rough plans to create one, but they got dropped immediately when the idea to merge Livna with other 3rd party repositories for Fedora came up. That lead to RPM Fusion being formed,which was announced one week ago.

Sharp eyes will have noticed that the announcement did not contain any information's regarding support for EL. That was on purpose – we focused on getting RPM Fusion running, hence the EL branch (which was planed right from the start) did not get as much love as the Fedora branch. But the EL branch is in the works and some of the most important RPM Fusion packages have been built for EL already. Xine and xine-lib-extras-freeworld for example are available in the testing repository already; the AMD and Nvidia drivers are still missing, just like mplayer and vlc. But all of them should hopefully get into the repositories over the next few weeks.

You can already help testing the RPM Fusion packages for EL repositories by enabling it with this command:

rpm -ivh http://download1.rpmfusion.org/free/el/updates/testing/5/x86_64/rpmfusion-free-release-5-0.1.noarch.rpm http://download1.rpmfusion.org/nonfree/el/updates/testing/5/x86_64/rpmfusion-nonfree-release-5-0.1.noarch.rpm

Some of the RPM Fusion packages for EL require bits from EPEL, hence it's necessary that you have EPEL configured and enabled as well – just like you had to enable Fedora Extras to use Livna in the Fedora Core days. Just remember that the RPM Fusion for EL repositories are still in the early stages. Also not that you for now need to enable epel-testing, as some of the RPM Fusion packages depend on packages that are currently in epel-testing. That requirement should hopefully vanish with the next testing -> stable move in EPEL. The RPM Fusion for EL repositories of course don't replace and packages from EL or EPEL.

Note that not all RPM Fusion contributors have interest in building their packages for EL (just like not all Fedora contributors participate in EPEL). Hence we could need some help and people with rpm packaging skills that help getting the RPM Fusion repositories for EL filled and maintained. If you are interested contact me in private or (preferred) join the mailing list and just start to help.

Second step of the transition from Livna to RPM Fusion: Moving F8 and F9 testing users over

After enabling the RPM Fusion repositories for users of Livna devel a few weeks ago we now do the next step to move livna users over to RPM Fusion: Enable RPM Fusion for those F8 and F9 users that have livna's testing repos enabled.

The way to do that is round about the same as in the devel branch earlier: I added the rpmfusion-release packages for RPM Fusion's free and nonfree repos to the livna repo for F8 and F9; afterwards I built new livna-release for F8 and F9 that went into livna-testing; those two livna-release packages track the two rpmfusion-release packages in with a hard dep. That way all users that installed livna properly (e.g. by installing the livna-release package) and enabled the testing repos will now get RPM Fusion enabled automatically.

Note, nearly all of livna's packages have been imported and build for RPM Fusion, but a few are still missing. So you should leave livna repos enabled for now if you want everything. Once all the packages have a new home we'll let the rpmfusion-nonfree-release package obsolete livna-release.

The plan is to move regular livna users of F8 and F9 over to RPM Fusion with the same trick sooner or later. But some things in RPM Fusion need to get brought in shape before we start considering that. But if you want you can already help by using and testing RPM Fusion for F8 and F9 by running this command:

rpm -ivh \
http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-stable.noarch.rpm \
http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-stable.noarch.rpm

It's even easier if you have livna enabled already:

yum install rpmfusion-free-release rpmfusion-nonfree-release

Please spread the news, to make sure all the docs in the internet get updated! tia!

P.S.: Just a reminder while at it: Some of you might have noticed already, the livna mailing lists (like freeworld{,-graphics}@livna.org) are dead since a few weeks; the hard disk in Anvil's mailman host died afaik (I don't know more details; sorry). But Livna will be superseded by RPM Fusion soon anyway, so simply use the RPM Fusion mailing lists from now on. They should serve well for the remaining time, as all the livna contributors should be subscribed there as well. Sorry for the trouble.