Dangers of Delay - How to prevent a Linux kernel fork through wider stakeholder participation

[ Thanks to Mohan Simham for this link. ]

The Dangers of Delay

How to prevent a Linux kernel fork through wider stakeholder participation

Synopsis: As the commercial market for Linux grows, vendors are coming under increasing pressure to deliver a definite roadmap to their customers with features, milestones and deadlines. Will the uneasy alliance between vendors and Free Software developers break down? Is there a way to reconcile the demand for a scheduled roadmap with the Free Software development philosophy that regards deadlines as undesirable?

It's not just us anymore

Most people today who have traditionally identified themselves as part of the "Linux community" understand that the high quality of Free Software stems from the fact that it is unencumbered by artificial deadlines thought up by a Marketing or Product Development department. As the community's leaders periodically remind us, software should be released when it is ready and not before. The contrast between this approach and the traditional commercial one of missed deadlines, buggy production releases and service packs is striking and instructive. This is indeed a revolutionary model of software engineering, where uncertainties of quality are traded off against uncertainties of time. It is also one that requires great courage and discipline to stick to. A patient and understanding user base is a major help to those in charge of Free Software development projects.

Today, the community of Linux users has grown far beyond what it was just a year ago, and it will become larger still as time goes by. Most new users have come to Linux attracted by its affordability and range of free applications. It is entirely likely that they have neither sympathy for the idealistic principles of Free Software nor an understanding of the quality advantages of Open Source. As proof of "mainstream" thinking affecting the Linux community, many articles have appeared that speak of "delays" in the Linux 2.4 kernel. These are angrily dismissed by hardcore Linux supporters with the reminder that there are no deadlines for Free Software, and so any talk of delays is meaningless.

But to many people, there is an expectation, realistic or not, that certain features will be delivered in a certain timeframe. When these expectations are not met, it is hard to convince them that no delay was involved. As we said before, there is a trade-off between uncertainties of time and uncertainties of quality. It is important to acknowledge both the existence of such a trade-off, and that not everyone will make the same choices. Newer users, especially corporate users, have a real need to plan their projects and budgets based on the likely availability of features. To many of them, a less-than-fully-debugged product released roughly on schedule followed by a service pack may well be an acceptable compromise.

Deriding such an attitude without acknowledging the very real compulsions behind it betrays a lack of understanding of the Linux community's changed demographic.

Accountability and the stakeholder model

Consider the fortunes of Steve Jobs and Rod Canion, the founders of Apple and Compaq respectively. Both were enormously respected visionaries who took their companies into the Fortune 500. Unfortunately for them, both were eased out by their boards when their style of functioning began to affect their companies' profitability and shareholder value (Steve Jobs has since returned in triumph to Apple, but Rod Canion hasn't been so lucky). It may seem ungrateful for a company to dump its founder, but the point is that any organisation or movement begins to affect the lives of a larger number of people as it grows, and all these people deserve a say in the way the organisation or movement is run. The whims of the original founder then become far less important than the interests of all the people they could affect.

All those whose lives are affected by an organisation or movement are called stakeholders. A truly democratic system will allow stakeholders to participate in decision-making, because the decisions affect them. This is why shareholders of companies have votes, and why employee unions often have representatives on the boards of companies. The American war of independence started because of a very simple principle -- "No taxation without representation". Indeed, the failure of the British government to recognise stakeholders other than themselves eventually led to the loss of all their colonies. In general, authority without accountability does not serve the needs of all stakeholders. A system of this sort cannot continue indefinitely without a "revolution" of some sort.

Today, there are many more stakeholders in the Linux community, because more people are affected by what happens with Linux. If we want our community to grow without fracturing, we must recognise and respect all of them.

• Hackers - They are the original founders of the movement. They are typically unpaid volunteers scratching their itch and their contributions deserve to be received with gratitude. If hackers feel alienated, they may stop contributing, which will be a loss to all stakeholders. It is important to point out that hackers are accountable to nobody.
• Amateur users - These are people who use Linux for the sheer love of working with the system. They are amateurs in the true sense of the term ("Amateur" comes from the Latin word for love - am). Some amateur users may contribute as well, through code or bug reports. They are not accountable to anyone either, and do not demand accountability from anyone. They are content with the community's support mechanisms to solve problems when they arise. They typically do not see themselves as having paid such significant amounts of money that they deserve to have someone accountable to them.
• Paying customers - These are people who recognise the value of Linux, but do not use Linux for the love of it. They use it to achieve specific goals, which could be cost savings, better customisability or vendor-independence. Such users are willing to pay to have their software supported in a guaranteed way, because downtime costs them even more. They desire accountability from those who provide them with Free Software.
• Linux solution providers - These are organisations that have sprung up largely to cater to the needs of paying customers. They are in the business for a profit, and are willing to accept accountability in return for a fee. So far, their relationship with the Free Software community has proven complementary. They provide the "guaranteed" support and services that paying customers want and the community itself does not provide. However, they do not control an important aspect of Free Software that paying customers want control over, -- new product features. Such control, especially in the kernel space, is jealously guarded by hackers. Solution providers face a tough balancing act as they try to satisfy their paying customers by assuming accountability on all matters important to them, while also maintaining good relationships with hackers.

As long as the Linux community consisted mostly of the first two types of stakeholders, life could go on without money or accountability entering the picture. But as the demographic shifts to include more stakeholders of the third and fourth groups, important changes begin to occur.

In the Free Software world, a typical response to feature requests is "If you need this feature, lend us a hand and write the code. If you can't code, just be patient." Development decisions, especially in the kernel, are made by a mechanism variously described as a meritocracy, benevolent dictatorship or enlightened despotism. Even if a group of users gets together and writes the features that they need, it may not be accepted into the kernel. What should those users do? Deny their own requirements, or run the risks of forking the kernel?

So what is likely to happen when the large mass of new users begins to exert pressure on vendors of Free Software? How long can commercial vendors keep up their polite deferral towards the community's leaders, who after all, have no accountability to their customers? The GPL has hitherto prevented forking not by forbidding it, but by taking away the commonest motive for forking, -- the desire to sell a proprietary variant. But with the emergence of a new motive for forking (greater control of one's destiny), the GPL may no longer be enough.

Case example 1:
The date is early 1999. An ISP is trying to decide on a platform, and their choices are Linux and Solaris. They require a journalled filesystem, file sizes above 2 GB and scalability up to 16 processors. Linux kernel 2.2 doesn't have these features, but kernel 2.4 looks like it will. The trouble is, they have no idea when it will be ready. What do they do? They need to take a decision now. If they have some confidence that 2.4 will roll around within (say) six months, they can go ahead and buy Intel-based servers, knowing that they will soon have the capabilities that they need. If they know that kernel 2.4 will definitely not be available for at least another six months, they must buy one or more Sun boxes. The problem here is that a stakeholder has no information, and no power to influence decisions like what non-essential features to take out of 2.4 to advance its release. Indeed, is there any objective way of determining what is "essential" and what is not, other than what Linus says?

Case example 2:
A large organisation calculates that the cost savings of moving their desktops from Windows to Linux comes to several million dollars a year. They find KDE 2 to be adequate for all their requirements except for the collaboration features offered by Microsoft Outlook. How can they get what they want? The answer is simple. If there is a demand, there will eventually be some commercial package available for Linux that meets this requirement. Since the organisation would probably be willing to pay for this package and still realize significant savings, this solution would be acceptable to all.

But what if a required feature needed to be in the kernel itself and Linus said no? Linus is not being mean or pig-headed when he refuses to incorporate features that he feels doesn't belong. He bases his decisions on his judgement of "good" and "bad" design. However, who is to say that the requests of a user group are not valid?

Case example 3:
A hardware solutions provider thinks there is a great market opportunity for multi-head workstations for the corporate market. A single machine can serve a number of monitors, enabling customer organisations to dramatically cut down on the number of machines they need to buy. The company has the in-house expertise to build the hardware and software. The trouble is, the modifications need to be made in the Linux kernel, and Linus doesn't believe that multi-head support belongs in the kernel. Should the vendor say "Oh, all right" and walk away from this market, or defy Linus by forking the kernel, earning the opprobrium of the Linux community in the process?

Today, all commercial vendors in the Linux space tread very carefully, anxious not to upset the "community". However, the traditional Linux community has and demands no accountability, whereas newer members care a lot about it. At some point in the future, such members will be the community, and the original group consisting of hackers and amateur users will be a minority. Will a vendor like IBM (which has now bet the farm on Linux) one day say to Linus, "Look here, Linus, you're doing a great job and all that, but we understand from our customers that they need XYZ in the kernel. We realise you don't like the idea, but if you don't want to do it, we'll have to do it ourselves and release it as an IBM kernel. We'll obey the GPL and release the source code for it, but we simply need to have that feature in there. We realise it will make our kernels incompatible, and we're sorry about that, but there's simply too much at stake."

How can the community tackle the shift in its centre of gravity without such a wrenching experience? Instead of wringing our hands as vendors like IBM reluctantly release forked kernels (with full source disclosure, of course), can we in the community devise a system that will prevent such forking while keeping alive the Free Software way that has delivered so much?

Implementing stakeholder democracy

The answer lies in enfranchising all stakeholders, in the true spirit of Free Software. For the system to succeed, the community leaders need to become more participative, and decisions need to be more democratic. It's hard for many to accept lay users voting on design decisions when kernel hackers are clearly more qualified, but then, Singapore's leaders feel the same way about running the country. What do ordinary citizens know about running a country? Leave it to the politicians. After all, we're doing such a great job.

In any case, users would probably not decide whether a global spin-lock is required or not. They would only say that SMP is important to them. Users would propose features such as this, and the entire community of stakeholders would vote to determine their relative priorities. Those that scratched hackers' itches would get done anyway, but the high-priority tasks too boring to attract volunteer developers would be funded by the stakeholders who need it done. Paid developers would then do the work.

A steering group would exist to ensure that the highest-priority tasks get folded into the next release at the earliest opportunity. To keep everyone honest, all proposals, voting, etc., would be done in the open. Delays to high-priority tasks would be readily visible, enabling stakeholders to either raise their stakes to encourage faster development, or downgrade their expectations. In either case, stakeholders would know exactly what they are getting, and gain a measure of control over their destiny.

We already have components of what we need to make such a system succeed. We only need to put them together.

• Organisations like CoSource and Collab.Net bring together users who want features and are willing to pay for their development, and developers who are willing to implement features for a fee. When enough demand for a feature exists (backed up by dollars), it attracts development effort. The source code itself remains free. It's only the act of development that is being paid for through this mechanism. That way, users don't have to wait for a hacker to scratch his itch before their desired features are implemented. The theory is that Free Software has use value rather than sale value, so the cost to a user of going without a feature is a reliable measure of that feature's importance. CoSource and Collab.Net use committed dollars as a surrogate for importance, and the approach seems to be working.
• Bug tracking software Bugzilla allows users to vote on the bugs they want fixed. The higher the number of votes for a bug, the greater the priority of the fix. This mechanism ensures that the largest groups of users have their requirements catered to first. (What about minority groups? They have to either fix their bugs themselves, pay someone to fix them, or wait their turn.)

We need a kernel forum where stakeholders of all kinds can propose their ideas and have them voted on by other stakeholders without having them shot down by persons unaccountable to them.

Conclusion

The rapidly changing demographic composition of the Linux community means that we have perhaps outgrown the ability of a benevolent dictatorship to look after the interests of all stakeholders. If nothing is done to recognise and include all stakeholders in a more participative process, a kernel fork may soon occur. To pre-empt this, the community's leaders (Linus Torvalds and others) must voluntarily adopt a system that gives stakeholders greater control of their own destiny. Ultimately, a democratic system is the only way to prevent a forking of the kernel.

Author's personal note: Nothing in the above article should be taken as a criticism of Linus Torvalds, for whom I have the highest regard and gratitude.

About the Author: Ganesh Prasad is a long-time Linux user who has not contributed a line of code to the Linux kernel, but worries about it all the same.