The sorry state of The Jem Report

Jem Matzan’s The Jem Report is running a so-called editorial by Radu-Cristian Fotescu (aka. Béranger) titled The sorry state of open source today. I say so-called, because it is more of a rant than an editorial: 26 pages long and not entirely coherent.

I won’t waste your time with a point-by-point rebuttal of this piece, not least because most of what he writes is pure opinion and interpretation. I don’t necessarily agree with it—I find him a little too radical and a little too confrontational—but he’s entitled to it.

(I do agree with his views on the differences between the GPL and the BSD license, but that’s neither here nor there)

What I take exception to are factual errors in his discussion of *BSD, and specifically of FreeBSD.

First, on page 7, he writes:

[members of] the *BSD family [are] either backed by 501(c)(3) non-profit organizations like The FreeBSD Foundation or the NetBSD Project, or the task of individuals like Theo de Raadt for OpenBSD and Matt Dillon for DragonFly BSD

This is inaccurate as far as FreeBSD goes. The FreeBSD Foundation was established because a recognized legal entity (which the FreeBSD Project itself is not) was required for three purposes:

The Foundation contributes to the Project in the sense that it owns some of the hardware that the Project uses, organizes and funds events such as the Developer Summits, which are held twice a year at BSDCan and EuroBSDCon, and occasionally funds development work.

This does not mean that the FreeBSD Project is backed by the FreeBSD Foundation. The Foundation does not control the Project in any way; it does not control the Project’s servers and repositories, it does not approve new committers, write roadmaps or schedule releases. The democratically elected Core Team does that.

This also invalidates the following quote from page 3:

even the FreeBSD foundation [sic] is an American subject

The implication is that since the FreeBSD Foundation is subject to American law, the FreeBSD Project is subject to American software patents.

Once again, the Foundation does not control FreeBSD. Moreover, the bulk of FreeBSD development today—at least the cutting-edge work—happens in Europe (especially Eastern Europe, but there are also strong contingents in Denmark, Italy and the UK) and Asia.

The NetBSD situation is different, as The NetBSD Foundation, Inc. appoints the NetBSD Core Group and various executive committees, and handles applications from new developers, who must sign a membership agreement with the Foundation before obtaining write access to the repositories.

The next objection I have relates to the following quotes from pages 10 and 22:

Maybe the first sign was when FreeBSD tried to mimic Linux and to be “more user-friendly”. It was rather wrong, and I can only hope they realized the mistake. The gradual recovery of the once legendary FreeBSD stability can be seen with 6.1 and 6.2. Hopefully the trend will not be reversed again.

Unfortunately, starting with 5.0, the quality and stability of FreeBSD have deteriorated, or maybe it was my hardware that was less supported. Compromises have been made to match the popularity of Linux, and the price to pay was rather high.

These two paragraphs are based on a completely incorrect understanding of what happened between FreeBSD 4 and FreeBSD 6. User friendliness, popularity, Linux envy: none of this ever played any part. I will attempt to briefly explain what actually happened.

FreeBSD first gained support for symmetric multiprocessing in 3.0. The approach chosen then was a simplistic one: the entire kernel was serialized under a so-called “giant lock”. This saved a lot of work, as there was no need to make the various kernel subsystems reentrant. It also meant that FreeBSD’s SMP performance was not very good, except for computationally intensive tasks (i.e. tasks which did not involve the kernel much), but it was better than nothing.

After FreeBSD 4 was released, work was started on a project called SMPng, which would replace the giant lock with a number of fine-grained locks, allowing separate kernel threads to run simultaneously on separate processors, performing separate tasks.

This is where two important mistakes were made.

Firstly, the release schedule was feature-bound rather then time-bound. This meant that FreeBSD 5 would be released “when it was done”, not “when it was time”. Consequently, there was no deadline for developers to focus on, and little effort was made to partition work into manageable pieces to avoid having the source tree in an unreleasable state for long periods of time.

Secondly, and most importantly, it was decided very early on to go for a complicated M:N scheduling model, based on a variant of scheduler activations called KSE. With KSE, a userland process with N threads (1 <= N) would receive M kernel threads (1 <= M <= N) on which to run. The userland threads library would be responsible for scheduling application threads on top of the kernel threads. Within the kernel, threads would be scheduled using a two-tier system involving KSE groups (groups of threads belonging to the same process) and individual threads.

Scheduler activations had academic backing and the M:N model had proved workable in Solaris, and NetBSD was moving in the same direction, but this decision was made before there was even proof-of-concept code to demonstrate its viability in FreeBSD.

(Ironically, around that time, Sun decided to abandon the M:N model in Solaris, and NetBSD is now moving to 1:1)

To cut a long story short, KSE was a disaster. It took years to implement, delaying other work which depended on it, and it never worked properly. FreeBSD 5 was going nowhere fast until a developer who until then had not been involved in either SMPng or KSE lost patience and implemented a 1:1 thread library on top of KSE. In the end, KSE was never even fully implemented, and it is now being slowly replaced with a pure 1:1 model.

So there you have it. FreeBSD 5 was the result of a failed experiment in M:N threading. FreeBSD 6 was an attempt to correct those mistakes, and FreeBSD 7 will dispose of KSE entirely and complete the fine-grained locking work that started with FreeBSD 5.

Finally, the FreeBSD Project learned an expensive lesson with FreeBSD 5. We now have a separate repository where experimental code can mature and high-risk changes can be tried out before entering CVS. We have also switched our release model to time-based releases rather than feature-based releases. If, as a release date approaches, we realize that a particular feature won’t be ready in time, we simply leave that feature out of the release rather than change our schedule.

16 thoughts on “The sorry state of The Jem Report”

  1. Dear Cat Lover,

    Your own post, albeit very informative, is full of contempt. And you’re wrong, both as an attitude and in your understanding.

    When I said that FreeBSD is backed by the Foundation, I have not implied that it is ‘controlled’ by it!

    Since when ‘backing’ somebody is ‘controlling’ it? To back someone is to provide support, assistance, some kind of help.

    back (transitive verb): to support or help by physical, moral, or financial assistance : UPHOLD : strengthen or encourage by aid or influence — often used with up ‘back a candidate for office’, ‘back up his son’.
    Webster’s Third New International Dictionary, Unabridged. Merriam-Webster, 2002. http://unabridged.merriam-webster.com

    So all your post is useless. It’s good for the information, but it has an aggressive attitude that does not honor you. Advocating FreeBSD by insulting people is not the right thing to do.

    And again: I have never said that it’s about “control”.

    As for being an American subject, it’s about the legal risks: should everything *BSD move to the U.K. (for an example), or at least to Canada, and the future would be safer.

  2. Wow. When I described you as “a little too radical and a little too confrontational”, I apparently didn’t know the half of it. Well, thanks for sharing, anyway.

  3. I’ll rebut your rebuttal =-)

    You’re absolutely correct about feature-based vs time-based being a
    problem. However, KSE was absolutely not the major nor the second major reason for the FreeBSD 5.x problems. 5.x releases suffered from the following problems that were much larger and much more immediate:

    – ULE and the modularized scheduler
    – PREEMPTION
    – ATA
    – UFS2
    – Immature locking model, too much Giant

    Now, I’ll entertain that the KSE development caused hurt feelings among some developers, but that was a professionalism issue, not a technical issue. I also do agree that M:N is a nice academic theory that has run into real-world roadblocks, and that FreeBSD seems to be better off in the end with 1:1 threads, just like most other OSes. But KSE was a stepping stone to get there; without it, who knows when we would have moved passed libc_r? It was a definitely a painful step, but it would
    have been much more painful to not have any alternatives to libc_r. I’m glad that the project and certain developers in it had the courage to do it AND to stick with it to resolve the tough problems.

    Scott

  4. I agree that there was plenty wrong with FreeBSD 5 besides KSE, but in hindsight KSE was where we went in over our heads, and the reason FreeBSD 5 was delayed for so long. If we had aimed for 1:1 from the beginning, we would have been able to focus our efforts on these other items.

    As far as ULE is concerned, I think the only mistake was to have it in GENERIC; if we had shipped with 4BSD instead, it wouldn’t have been an issue.

  5. I can state definitively that no 5.x releases were held up by any significant amount due to KSE. Having a working libkse was actually a nice surprise for 5.1, and it certainly wasn’t expected after 5.0 came out. On the other hand, ATA caused 3 months of turmoil with 5.2 and forced us to do 5.2.1. PREEMPTION and ULE caused 5 months of incredible pain and delay with 5.3, and if the author of KSE hadn’t stepped in to help fix it (the authors of PREEMPTION and ULE helped very little, btw), it would have been a long-lasting blow to the project. It wasn’t just the fault of ULE being made the default; the modularization of the scheduler framework, which was done to support ULE, made 4BSD unstable as well. I would have released 5.3 much earlier if merely disabling ULE had fixed the problems. KSE had nothing to do with it.

  6. I’m not talking about delays between individual 5.x releases, I’m talking about the time it took to get 5.0 out of -CURRENT. The first publicly announced release date for 5.0 was November, 2001, and that was a pessimistic estimate – ISTR we originally aimed for early 2001. In the end, we didn’t release 5.0 until January, 2003, and even that was barely usable, as you yourself admitted in the release announcement.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.