In Context: Linuxcare's Experience in Ramping Up a Bazaar Business ModelOct 12, 2000, 09:17 (0 Talkback[s])
(Other stories by John Wolley)
Linuxcare has boldly gone where no one else had gone before in creating an open source business model to offer services and only services for Linux and open source software. Although their experience is unique, it also illustrates the type of problems faced by any organization that attempts to launch an open source business model in any area where no one else has done it before.
by John Wolley, Linux Today
By now the world has gotten over the initial disbelief that businesses can make money off of software that's given away for free. It's now universally accepted in the mainstream press that money can be made with Linux by providing services -- the doubts now revolve around how much money "Linux companies" can make and whether they can remain independent.
While it seems that every "Linux company" now counts services as one of its key revenue opportunities, Linuxcare is the quintessential service company for Linux. But what exactly are the services that Linuxcare offers and what is it like to provide these services in an open source business model? Linuxcare started out offering "tech support", but today the vast majority of its revenue comes from the higher value added area that's termed "professional services" -- the story of how this shift occurred is really interesting.
I had been talking with various people at Linuxcare for nearly a year about doing an "Inside Linuxcare" story, detailing the major problems, successes, and surprises that they've faced in the process of burning in a services-based, open source business model. (Actually, the credit goes to Matthew Cunningham, Linuxare's PR guy up until the big layoff, who suggested it at a Silicon Valley LUG [Linux Users Group] meeting around Christmas of '99.) Recently I had an opportunity to interview Linuxcare founders, David L. Sifry, Arthur Tyde, and David LaDuke, and ask these questions first-hand.
The story they tell is of course unique to Linuxcare. But it also illustrates the general problem that's faced by any organization that attempts to launch an open source business model in any area where no one else has done it before. In the Cathedral & Bazaar metaphor, inevitably our ideas about how such a business will work in the Bazaar are colored by our experience about how such businesses have worked in the Cathedral. We know that everything rides on providing services that meet customers' needs. But when the customers show up and try out the services offered, they're likely to discover that they had needs they didn't even know about!
The Original Idea -- "An 800-number for
Now Sifry and Tyde, along with most of the initial Linuxcare staff they recruited, mostly from BALUG, had some serious experience in Cathedral-style tech support, so they knew that tech support-related services would involve a lot more than just manning an 800-number. For example, with proprietary software, in the process of helping customers with the problems they encounter, tech support routinely uncovers bugs and design flaws and passes this information on to the quality assurance (QA) folks. The QA people perform additional tests to determine the full range of conditions over which the problems occur and pass this information on to the appropriate developers. Hopefully the developers eventually make changes in the code that incorporate the customer feedback.
Tech support also is the primary feedback channel from customers: general feedback about what customers like and dislike, complaints about features that are difficult to use, requests for "enhancements" that would make the software much more useful and valuable. And by analyzing the patterns in customer calls, tech support can identify software problems that individual customers don't even articulate. For example, if a high percentage of customers are screwing up on the same step in an install process, tech support can identify this install step as a serious problem, even if each individual customer sees it as their own mistake (e.g., misreading a prompt) -- and tech support may be able to recommend obvious solutions, like changing the wording on a prompt.
Tech support services can hit a serious snag in the Cathedral model when multiple vendors are involved -- the tech support staff working for each vendor investigates the problem separately, determines to the best of their ability that it is not originating in their product, and so "pass the buck" to the other vendor(s).
Anticipating how tech support-related services would work in the Bazaar, Sifry and Tyde saw several obvious extensions over the Cathedral model which would present both serious challenges and significant revenue opportunities. First, since they would be offering their services independent of any vendor, they would have to be able to diagnose problems and pinpoint the solutions, no matter how many different vendor products were involved. Second, they could go ahead and fix the problems that are identified, because they would be dealing with open source -- and it was expected that this would expand into services like customization, porting, migration, and integration. Additionally, they foresaw a demand for independent certification that a given product, like a laptop PC, works well with another product, like a given vendor's Linux distro. And then, with involvement in all these other areas, it seemed logical that they could provide education and training as well.
Who are the Paying Customers? -- Large
The typical Fortune 500 corporation, which is only concerned with their own internal Linux tech support-related issues, still hasn't seen the value that Linuxcare services could offer them. This may change dramatically if these corporations start deploying Linux and other open source software on a large scale.
Not surprisingly, most of the vendors who have contracted with Linuxcare have started out with the absolute minimum level of services they thought they could get by with. They have typically looked at what Linuxcare can provide and what it would cost, and said "We can do that cheaper internally," only to return six months later, ready to sign up Linuxcare for a major portion of their Linux tech support function.
Most vendors initially sign up for per-incident support. They monitor what they're getting until they determine that the basic tech support is indispensable and the product feedback is extremely valuable -- then they switch to a monthly minimum fee, which is far more cost effective for any significant volume of calls.
How to Make it Scale? -- the Linuxcare
Traditionally, Linux users have relied heavily on newsgroups to get answers to their tech support questions. Instead of having end users post questions to a newsgroup, then waiting to see what sort of replies came back, Linuxcare quickly got a "knowledgebase" up and running and set up a query interface that allows end users to get answers to a lot of the simpler problems that come up. Of course, the bigger and more comprehensive the knowledgebase is, the more likely that an end user can find a solution to any particular problem.
The knowledgebase concept also figures prominently in most Cathedral tech support models (MS has one). How does Linuxcare's Bazaar model knowledgebase differ from the Cathedral models? Its vendor independence is probably the biggest difference -- a knowledgebase set up by a proprietary vendor deals primarily with that vendor's products, while Linuxcare's knowledgebase deals with just about everything (hardware as well as software) that a user of any particular Linux distro might encounter.
The multi-vendor nature of Linuxcare's knowledgebase led to a use that wouldn't even be needed in a proprietary, single-vendor environment. As the knowledgebase began to expand and function nicely in its primary role, a second "deltabase" function became obvious: delineating the differences between any two environments. This allows developers who have a software product configured for a particular distro -- like Red Hat -- to get from the knowledgebase just the differences between that environment and another one they want the software to work in -- like Red Flag Linux.
Another interesting feature that I haven't personally encountered in knowledgebases for the largest proprietary software vendors, Microsoft and Oracle, is the way you set up a user profile that is implicitly factored into your query -- when you register, you specify all the details of your hardware and software environment, and those details are automatically used to filter the results of your queries so you are only shown answers that are relevant to your environment.
Keeping Those Customers Satisfied
How satisfied are Linuxcare's vendor customers and their end users with Linuxcare's services? Well, maybe Linuxcare has an edge here, because their performance is typically being compared with the notoriously bad tech support-related services that people are used to receiving under the Cathedral model. But Linuxcare's ongoing surveys of end user satisfaction continually show that the "people with the problems" are delighted with the quality of the service they receive, including the speed with which they receive it.
The large vendors who are paying Linuxcare to provide the services are typically having their expectations about the value of Linuxcare's services to their businesses "dramatically exceeded." Linuxcare's corporate customers are steadily expanding the scope of work they are having Linuxcare do, and new corporate customers are signing up. At the beginning of this year, several financial analysts noted that Linuxcare's customer base was still fairly small and thus revenues could dip abruptly if any one of their biggest customers were to cancel their contracts. Sifry, Tyde, and LaDuke say they are not concerned about such a possibility today -- yes, the customer base has expanded, but the feedback they are getting from their major customers is so positive that they regard the possibility of a customer defection as "extremely unlikely."
If the August announcement of Linuxcare's latest $30 million round of funding is any guide, its corporate customers are very happy indeed: Dell (through Dell Ventures), Motorola and Sun Microsystems, major Linuxcare customers who contributed to earlier rounds of funding, came back to contribute in this round. The funding will keep Linuxcare on track in its move toward the IPO milestone that Red Hat, Cobalt Networks, and VA Linux achieved in '99. Other major customers that keep coming back for more Linuxcare services include Compaq Computer Corp., Fujitsu Siemens Computers, Hewlett-Packard, and IBM.
Isn't the Distro Complexity a Nightmare to Deal With? --
Tyde's answer: No. Why? "There's no such thing as a 'standard' Linux distribution." Remember that Linuxcare's customers are mostly large vendors. In practice, these customers typically take a "standard" distribution like Red Hat and customize it to meet their specific needs. Thus Linuxcare has to keep track of all the details of each customer's distro customizations anyway. After gearing up to have the Linuxcare knowledgebase accommodate that level of complexity, extending the scope to encompass the other major distros didn't really complicate things that much more.
Linuxcare does draw a line at kernel customizations. All the information in the knowledgebase relates to unmodified versions of the Linux kernel, as released by the kernel team. They can provide services in support of software that involves kernel modifications, such as Beowulf clustering, but it's not supported by the knowledgebase. It relies on the expertise of individual technicians, tends to be extremely labor-intensive, and hence tends to be more expensive to provide.
Some Examples of the Surprises
It was anticipated that customers would initially sign up for the basic tech support service -- farming out tech support to Linuxcare -- and then gradually try out additional services as they saw the benefits of the basic service and gained confidence in Linuxcare. What they found was that customers were eager to sign up for professional services (customization, porting, integration, migration), product certification, and training much earlier than they had expected.
Below are some examples of services that Linuxcare customers requested fairly soon after they had signed up for "basic" tech support. These are services that the original Linuxcare business plan envisioned offering, just not so soon.
Linuxcare's experience with ramping up their Bazaar business model has validated -- with a vengeance -- the basic open source concept about how money should be made with open source software. The real value lies in services that make the free code do exactly what people need it to do, within the specific technical environment that they need it to do it. The availability of such services is so revolutionary, in comparison with what people are used to settling for with proprietary software, that customers really are eager to pay what the services cost -- it's "a bargain at twice the price!" For open source businesses, like Linuxcare, that want to sell such services, the challenge is gearing up to be able to provide all the services that their customers will ask for.