In Context: Linuxcare's Experience in Ramping Up a Bazaar Business Model

Linuxcare has boldly gone where no one else had gone before in creating an open source business model to offer services and only services for Linux and open source software. Although their experience is unique, it also illustrates the type of problems faced by any organization that attempts to launch an open source business model in any area where no one else has done it before.

by John Wolley, Linux Today

By now the world has gotten over the initial disbelief that businesses can make money off of software that's given away for free. It's now universally accepted in the mainstream press that money can be made with Linux by providing services -- the doubts now revolve around how much money "Linux companies" can make and whether they can remain independent.

While it seems that every "Linux company" now counts services as one of its key revenue opportunities, Linuxcare is the quintessential service company for Linux. But what exactly are the services that Linuxcare offers and what is it like to provide these services in an open source business model? Linuxcare started out offering "tech support", but today the vast majority of its revenue comes from the higher value added area that's termed "professional services" -- the story of how this shift occurred is really interesting.

I had been talking with various people at Linuxcare for nearly a year about doing an "Inside Linuxcare" story, detailing the major problems, successes, and surprises that they've faced in the process of burning in a services-based, open source business model. (Actually, the credit goes to Matthew Cunningham, Linuxare's PR guy up until the big layoff, who suggested it at a Silicon Valley LUG [Linux Users Group] meeting around Christmas of '99.) Recently I had an opportunity to interview Linuxcare founders, David L. Sifry, Arthur Tyde, and David LaDuke, and ask these questions first-hand.

The story they tell is of course unique to Linuxcare. But it also illustrates the general problem that's faced by any organization that attempts to launch an open source business model in any area where no one else has done it before. In the Cathedral & Bazaar metaphor, inevitably our ideas about how such a business will work in the Bazaar are colored by our experience about how such businesses have worked in the Cathedral. We know that everything rides on providing services that meet customers' needs. But when the customers show up and try out the services offered, they're likely to discover that they had needs they didn't even know about!

The Original Idea -- "An 800-number for Linux"
To make a long story short, Sifry and Tyde had gotten together through the Bay Area LUG (BALUG, which Tyde had started) and had an idea for a VPN (virtual private network) server business that they were trying to get funded. When they would talk to a venture capital group about funding, they were finding that there was little interest in funding a business that would be competing with networking giant Cisco; but whenever Linux came up around the edges of the conversation, they were finding a lot of interest. Eventually, Sifry, Tyde, and LaDuke got the message -- one fateful Friday in August, 1998, the "light went on" and they got the original idea for Linuxcare, "an 800-number for Linux."

Now Sifry and Tyde, along with most of the initial Linuxcare staff they recruited, mostly from BALUG, had some serious experience in Cathedral-style tech support, so they knew that tech support-related services would involve a lot more than just manning an 800-number. For example, with proprietary software, in the process of helping customers with the problems they encounter, tech support routinely uncovers bugs and design flaws and passes this information on to the quality assurance (QA) folks. The QA people perform additional tests to determine the full range of conditions over which the problems occur and pass this information on to the appropriate developers. Hopefully the developers eventually make changes in the code that incorporate the customer feedback.

Tech support also is the primary feedback channel from customers: general feedback about what customers like and dislike, complaints about features that are difficult to use, requests for "enhancements" that would make the software much more useful and valuable. And by analyzing the patterns in customer calls, tech support can identify software problems that individual customers don't even articulate. For example, if a high percentage of customers are screwing up on the same step in an install process, tech support can identify this install step as a serious problem, even if each individual customer sees it as their own mistake (e.g., misreading a prompt) -- and tech support may be able to recommend obvious solutions, like changing the wording on a prompt.

Tech support services can hit a serious snag in the Cathedral model when multiple vendors are involved -- the tech support staff working for each vendor investigates the problem separately, determines to the best of their ability that it is not originating in their product, and so "pass the buck" to the other vendor(s).

Anticipating how tech support-related services would work in the Bazaar, Sifry and Tyde saw several obvious extensions over the Cathedral model which would present both serious challenges and significant revenue opportunities. First, since they would be offering their services independent of any vendor, they would have to be able to diagnose problems and pinpoint the solutions, no matter how many different vendor products were involved. Second, they could go ahead and fix the problems that are identified, because they would be dealing with open source -- and it was expected that this would expand into services like customization, porting, migration, and integration. Additionally, they foresaw a demand for independent certification that a given product, like a laptop PC, works well with another product, like a given vendor's Linux distro. And then, with involvement in all these other areas, it seemed logical that they could provide education and training as well.

Who are the Paying Customers? -- Large Vendors
One of the key decisions Linuxcare made early on was about who the paying customer would be: It was clear that it would be far simpler and easier to conclude agreements with major commercial vendors than to try to sign up their thousands of end user customers individually. In effect, Linuxcare offers major vendors the ability to "farm out" all or part of their tech support functions to Linuxcare. The vendor gets to pick and choose what services they want to pay for initially, and they can add (or drop) services whenever they want to, all without having to do their own staffing and training.

The typical Fortune 500 corporation, which is only concerned with their own internal Linux tech support-related issues, still hasn't seen the value that Linuxcare services could offer them. This may change dramatically if these corporations start deploying Linux and other open source software on a large scale.

Not surprisingly, most of the vendors who have contracted with Linuxcare have started out with the absolute minimum level of services they thought they could get by with. They have typically looked at what Linuxcare can provide and what it would cost, and said "We can do that cheaper internally," only to return six months later, ready to sign up Linuxcare for a major portion of their Linux tech support function.

Most vendors initially sign up for per-incident support. They monitor what they're getting until they determine that the basic tech support is indispensable and the product feedback is extremely valuable -- then they switch to a monthly minimum fee, which is far more cost effective for any significant volume of calls.

How to Make it Scale? -- the Linuxcare Knowledgebase
One of the biggest initial problems that Linuxcare had to solve to profitably offer tech support was "how to make it scale?" That is, fielding customer questions over the phone, one-on-one, is very labor-intensive and could quickly become prohibitively expensive. So the Linuxcare founders set out to leverage the traditional strengths of Linux to best advantage.

Traditionally, Linux users have relied heavily on newsgroups to get answers to their tech support questions. Instead of having end users post questions to a newsgroup, then waiting to see what sort of replies came back, Linuxcare quickly got a "knowledgebase" up and running and set up a query interface that allows end users to get answers to a lot of the simpler problems that come up. Of course, the bigger and more comprehensive the knowledgebase is, the more likely that an end user can find a solution to any particular problem.

The knowledgebase concept also figures prominently in most Cathedral tech support models (MS has one). How does Linuxcare's Bazaar model knowledgebase differ from the Cathedral models? Its vendor independence is probably the biggest difference -- a knowledgebase set up by a proprietary vendor deals primarily with that vendor's products, while Linuxcare's knowledgebase deals with just about everything (hardware as well as software) that a user of any particular Linux distro might encounter.

The multi-vendor nature of Linuxcare's knowledgebase led to a use that wouldn't even be needed in a proprietary, single-vendor environment. As the knowledgebase began to expand and function nicely in its primary role, a second "deltabase" function became obvious: delineating the differences between any two environments. This allows developers who have a software product configured for a particular distro -- like Red Hat -- to get from the knowledgebase just the differences between that environment and another one they want the software to work in -- like Red Flag Linux.

Another interesting feature that I haven't personally encountered in knowledgebases for the largest proprietary software vendors, Microsoft and Oracle, is the way you set up a user profile that is implicitly factored into your query -- when you register, you specify all the details of your hardware and software environment, and those details are automatically used to filter the results of your queries so you are only shown answers that are relevant to your environment.

Keeping Those Customers Satisfied
How well is Linuxcare's knowledgebase working now? Some 80% of user questions are now being answered by user queries against the knowledgebase, without ever talking to a live support person. I walked through aisles of cubicles of people tasked with answering the phones and saw several people hunched over their PCs answering email, but hardly anyone answering phones. So when a problem does escalate into a live phone call, there's a Linuxcare staffer ready to jump on it. According to Sifry: "Linuxcare typically resolves the toughest, 'level 3' problems in under three minutes. That's faster than most tech support operations can pick up the phone!"

How satisfied are Linuxcare's vendor customers and their end users with Linuxcare's services? Well, maybe Linuxcare has an edge here, because their performance is typically being compared with the notoriously bad tech support-related services that people are used to receiving under the Cathedral model. But Linuxcare's ongoing surveys of end user satisfaction continually show that the "people with the problems" are delighted with the quality of the service they receive, including the speed with which they receive it.

The large vendors who are paying Linuxcare to provide the services are typically having their expectations about the value of Linuxcare's services to their businesses "dramatically exceeded." Linuxcare's corporate customers are steadily expanding the scope of work they are having Linuxcare do, and new corporate customers are signing up. At the beginning of this year, several financial analysts noted that Linuxcare's customer base was still fairly small and thus revenues could dip abruptly if any one of their biggest customers were to cancel their contracts. Sifry, Tyde, and LaDuke say they are not concerned about such a possibility today -- yes, the customer base has expanded, but the feedback they are getting from their major customers is so positive that they regard the possibility of a customer defection as "extremely unlikely."

If the August announcement of Linuxcare's latest $30 million round of funding is any guide, its corporate customers are very happy indeed: Dell (through Dell Ventures), Motorola and Sun Microsystems, major Linuxcare customers who contributed to earlier rounds of funding, came back to contribute in this round. The funding will keep Linuxcare on track in its move toward the IPO milestone that Red Hat, Cobalt Networks, and VA Linux achieved in '99. Other major customers that keep coming back for more Linuxcare services include Compaq Computer Corp., Fujitsu Siemens Computers, Hewlett-Packard, and IBM.

Isn't the Distro Complexity a Nightmare to Deal With? -- No
People familiar with Cathedral-style tech support-related services, where the focus is on the software products from a single vendor, tend to shake their heads in disbelief when they first hear that Linuxcare supports all major Linux distributions. Basically, if there is enough demand for a particular distro in business use, as long as the distro uses a standard Linux kernel, Linuxcare will add it to the their knowledgebase. In addition, Linuxcare supports the most common applications that run on the different distros, such as Apache, KDE and GNOME, and StarOffice. Isn't this an absolute nightmare of complexities to deal with? -- Won't they eventually be forced to pull back and concentrate on the top five or so most popular "standard" Linux distributions?

Tyde's answer: No. Why? "There's no such thing as a 'standard' Linux distribution." Remember that Linuxcare's customers are mostly large vendors. In practice, these customers typically take a "standard" distribution like Red Hat and customize it to meet their specific needs. Thus Linuxcare has to keep track of all the details of each customer's distro customizations anyway. After gearing up to have the Linuxcare knowledgebase accommodate that level of complexity, extending the scope to encompass the other major distros didn't really complicate things that much more.

Linuxcare does draw a line at kernel customizations. All the information in the knowledgebase relates to unmodified versions of the Linux kernel, as released by the kernel team. They can provide services in support of software that involves kernel modifications, such as Beowulf clustering, but it's not supported by the knowledgebase. It relies on the expertise of individual technicians, tends to be extremely labor-intensive, and hence tends to be more expensive to provide.

Some Examples of the Surprises
The Linuxcare founders had enough background in Cathedral-style tech support-related services, and enough imagination to foresee the Bazaar-style opportunities, so that they were able to accurately anticipate the general outlines of the services that their customers (large vendors) would want. What surprised them was the timing.

It was anticipated that customers would initially sign up for the basic tech support service -- farming out tech support to Linuxcare -- and then gradually try out additional services as they saw the benefits of the basic service and gained confidence in Linuxcare. What they found was that customers were eager to sign up for professional services (customization, porting, integration, migration), product certification, and training much earlier than they had expected.

Below are some examples of services that Linuxcare customers requested fairly soon after they had signed up for "basic" tech support. These are services that the original Linuxcare business plan envisioned offering, just not so soon.

• Dell, one of Linuxcare's biggest customers, shipped a Linux workstation with an older version of a video card driver that turned out to be buggy (it was the version that came with Red Hat 6.0). Linuxcare detected the problem because several Dell customers had called in. When you ran X windows, the clock slowed down and ran at 1/8 speed, so it took 8 minutes to do 1 minute's worth of work. After Linuxcare gave Dell the feedback on the problem, Dell sent a workstation to Linuxcare, which identified the problem in the video card and got Dell the correct driver. Dell burned a new version of Red Hat with the updated driver to fix the factory image.
• Netaid/KPMG called Linuxcare because their SSL performance was not fast enough. They were getting 10-15 tps (transactions per second) per box, which was way too slow. They needed a sustained 700-1000 tps, but were balking at having to pay for 70 boxes to get it. Linuxcare tested three different solutions and determined that Rainbow was the best one even though the Linux crypto-card driver was buggy. Linuxcare rewrote the kernel driver and Netaid/KPMG was able to get the SSL performance they needed by purchasing only three servers, and one of those was just for failover.
• A major hardware vendor wanted to roll out their new laptop. They came to Linuxcare to do a custom Linux build because in the standard Caldera 2.4 build most notebook related features didn't work right -- they couldn't get full functionality of some important features like power management, sound card, video card, and networking capabilities. Linuxcare got Caldera Linux up and running with full functionality. They even got the sound card to suspend and resume better than it did on Windows, which the hardware vendor said could not be done. On Windows it would suspend and then give you a blue screen, which they said was due to a BIOS bug, but Linuxcare went ahead and patched the Linux kernel to work around it. The hardware vendor's acceptance team was "blown away" by the results that Linuxcare provided.

Linuxcare's experience with ramping up their Bazaar business model has validated -- with a vengeance -- the basic open source concept about how money should be made with open source software. The real value lies in services that make the free code do exactly what people need it to do, within the specific technical environment that they need it to do it. The availability of such services is so revolutionary, in comparison with what people are used to settling for with proprietary software, that customers really are eager to pay what the services cost -- it's "a bargain at twice the price!" For open source businesses, like Linuxcare, that want to sell such services, the challenge is gearing up to be able to provide all the services that their customers will ask for.

Related Stories:

• Linux.com: The Future of Linuxcare - The Dave Sifry Interview(Oct 02, 2000)
• European Unix Platform (EUP): Book Review: The Cathedral and the Bazaar(Sep 30, 2000)
• CNET News.com: Sun takes a shine to Linuxcare's storage software(Aug 05, 2000)
• CNET News.com: Linuxcare looks to future with new financing(Aug 02, 2000)
• LinuxWorld (AU): Linuxcare Founder [David Sifry] Talks Open Source(Jul 07, 2000)
• Enterprise Linux Today: Linuxcare and Netaid(Jun 22, 2000)
• TechWeb: Linuxcare To Fine Tune Linux For Thinkpads(Jun 09, 2000)
• SunWorld.com: The truth about open source software [Book Reviews: The Cathedral and the Bazaar](May 10, 2000)
• Slashdot: The Cathedral and the Bazaar [Book Review](Nov 22, 1999)