Linuxcare has boldly gone where no one else had gone before
in creating an open source business model to offer services and
only services for Linux and open source software. Although their
experience is unique, it also illustrates the type of problems
faced by any organization that attempts to launch an open
source business model in any area where no one else has done it
before.
by John Wolley, Linux
Today
By now the world has gotten over the initial disbelief that
businesses can make money off of software that’s given away for
free. It’s now universally accepted in the mainstream press that
money can be made with Linux by providing services — the
doubts now revolve around how much money “Linux companies” can make
and whether they can remain independent.
While it seems that every “Linux company” now counts
services as one of its key revenue opportunities, Linuxcare is the
quintessential service company for Linux. But what exactly are the
services that Linuxcare offers and what is it like to provide these
services in an open source business model? Linuxcare started out
offering “tech support”, but today the vast majority of its revenue
comes from the higher value added area that’s termed “professional
services” — the story of how this shift occurred is really
interesting.
I had been talking with various people at Linuxcare for nearly a
year about doing an “Inside Linuxcare” story, detailing the major
problems, successes, and surprises that they’ve faced in the
process of burning in a services-based, open source business model.
(Actually, the credit goes to Matthew Cunningham, Linuxare’s PR guy
up until the big layoff, who suggested it at a Silicon Valley LUG
[Linux Users Group] meeting around Christmas of ’99.) Recently I
had an opportunity to interview Linuxcare founders, David L. Sifry,
Arthur Tyde, and David LaDuke, and ask these questions
first-hand.
The story they tell is of course unique to Linuxcare. But it
also illustrates the general problem that’s faced by any
organization that attempts to launch an open source business model
in any area where no one else has done it before. In the Cathedral
& Bazaar metaphor, inevitably our ideas about how such a
business will work in the Bazaar are colored by our experience
about how such businesses have worked in the Cathedral. We
know that everything rides on providing services that meet
customers’ needs. But when the customers show up and try out the
services offered, they’re likely to discover that they had needs
they didn’t even know about!
The Original Idea — “An 800-number for
Linux”
To make a long story short, Sifry and Tyde had gotten together
through the Bay Area LUG (BALUG, which Tyde had started) and had an
idea for a VPN (virtual private network) server business that they
were trying to get funded. When they would talk to a venture
capital group about funding, they were finding that there was
little interest in funding a business that would be competing with
networking giant Cisco; but whenever Linux came up around the edges
of the conversation, they were finding a lot of interest.
Eventually, Sifry, Tyde, and LaDuke got the message — one fateful
Friday in August, 1998, the “light went on” and they got the
original idea for Linuxcare, “an 800-number for Linux.”
Now Sifry and Tyde, along with most of the initial Linuxcare
staff they recruited, mostly from BALUG, had some serious
experience in Cathedral-style tech support, so they knew
that tech support-related services would involve a lot more than
just manning an 800-number. For example, with proprietary software,
in the process of helping customers with the problems they
encounter, tech support routinely uncovers bugs and design flaws
and passes this information on to the quality assurance (QA) folks.
The QA people perform additional tests to determine the full range
of conditions over which the problems occur and pass this
information on to the appropriate developers. Hopefully the
developers eventually make changes in the code that incorporate the
customer feedback.
Tech support also is the primary feedback channel from
customers: general feedback about what customers like and dislike,
complaints about features that are difficult to use, requests for
“enhancements” that would make the software much more useful and
valuable. And by analyzing the patterns in customer calls,
tech support can identify software problems that individual
customers don’t even articulate. For example, if a high percentage
of customers are screwing up on the same step in an install
process, tech support can identify this install step as a serious
problem, even if each individual customer sees it as their own
mistake (e.g., misreading a prompt) — and tech support may be able
to recommend obvious solutions, like changing the wording on a
prompt.
Tech support services can hit a serious snag in the Cathedral
model when multiple vendors are involved — the tech support staff
working for each vendor investigates the problem separately,
determines to the best of their ability that it is not originating
in their product, and so “pass the buck” to the other
vendor(s).
Anticipating how tech support-related services would work in the
Bazaar, Sifry and Tyde saw several obvious extensions over the
Cathedral model which would present both serious challenges and
significant revenue opportunities. First, since they would be
offering their services independent of any vendor, they would
have to be able to diagnose problems and pinpoint the
solutions, no matter how many different vendor products were
involved. Second, they could go ahead and fix the problems
that are identified, because they would be dealing with open source
— and it was expected that this would expand into services like
customization, porting, migration, and integration. Additionally,
they foresaw a demand for independent certification that a given
product, like a laptop PC, works well with another product, like a
given vendor’s Linux distro. And then, with involvement in all
these other areas, it seemed logical that they could provide
education and training as well.
Who are the Paying Customers? — Large
Vendors
One of the key decisions Linuxcare made early on was about who the
paying customer would be: It was clear that it would be
far simpler and easier to conclude agreements with major commercial
vendors than to try to sign up their thousands of end user
customers individually. In effect, Linuxcare offers major vendors
the ability to “farm out” all or part of their tech support
functions to Linuxcare. The vendor gets to pick and choose what
services they want to pay for initially, and they can add (or drop)
services whenever they want to, all without having to do their own
staffing and training.
The typical Fortune 500 corporation, which is only concerned
with their own internal Linux tech support-related issues, still
hasn’t seen the value that Linuxcare services could offer them.
This may change dramatically if these corporations start deploying
Linux and other open source software on a large scale.
Not surprisingly, most of the vendors who have contracted with
Linuxcare have started out with the absolute minimum level
of services they thought they could get by with. They have
typically looked at what Linuxcare can provide and what it would
cost, and said “We can do that cheaper internally,” only to return
six months later, ready to sign up Linuxcare for a major portion of
their Linux tech support function.
Most vendors initially sign up for per-incident support. They
monitor what they’re getting until they determine that the basic
tech support is indispensable and the product feedback is extremely
valuable — then they switch to a monthly minimum fee, which is far
more cost effective for any significant volume of calls.
How to Make it Scale? — the Linuxcare
Knowledgebase
One of the biggest initial problems that Linuxcare had to solve to
profitably offer tech support was “how to make it scale?” That is,
fielding customer questions over the phone, one-on-one, is very
labor-intensive and could quickly become prohibitively expensive.
So the Linuxcare founders set out to leverage the traditional
strengths of Linux to best advantage.
Traditionally, Linux users have relied heavily on newsgroups to
get answers to their tech support questions. Instead of having end
users post questions to a newsgroup, then waiting to see what sort
of replies came back, Linuxcare quickly got a “knowledgebase” up
and running and set up a query interface that allows end users to
get answers to a lot of the simpler problems that come up. Of
course, the bigger and more comprehensive the knowledgebase is, the
more likely that an end user can find a solution to any particular
problem.
The knowledgebase concept also figures prominently in most
Cathedral tech support models (MS has one). How does Linuxcare’s
Bazaar model knowledgebase differ from the Cathedral models? Its
vendor independence is probably the biggest difference — a
knowledgebase set up by a proprietary vendor deals primarily with
that vendor’s products, while Linuxcare’s knowledgebase deals with
just about everything (hardware as well as software) that a user of
any particular Linux distro might encounter.
The multi-vendor nature of Linuxcare’s knowledgebase led to a
use that wouldn’t even be needed in a proprietary, single-vendor
environment. As the knowledgebase began to expand and function
nicely in its primary role, a second “deltabase” function became
obvious: delineating the differences between any two environments.
This allows developers who have a software product configured for a
particular distro — like Red Hat — to get from the knowledgebase
just the differences between that environment and another one they
want the software to work in — like Red Flag Linux.
Another interesting feature that I haven’t personally
encountered in knowledgebases for the largest proprietary software
vendors, Microsoft and Oracle, is the way you set up a user profile
that is implicitly factored into your query — when you register,
you specify all the details of your hardware and software
environment, and those details are automatically used to filter the
results of your queries so you are only shown answers that are
relevant to your environment.
Keeping Those Customers Satisfied
How well is Linuxcare’s knowledgebase working now? Some 80% of user
questions are now being answered by user queries against the
knowledgebase, without ever talking to a live support person. I
walked through aisles of cubicles of people tasked with answering
the phones and saw several people hunched over their PCs answering
email, but hardly anyone answering phones. So when a problem
does escalate into a live phone call, there’s a Linuxcare
staffer ready to jump on it. According to Sifry: “Linuxcare
typically resolves the toughest, ‘level 3’ problems in under three
minutes. That’s faster than most tech support operations can pick
up the phone!”
How satisfied are Linuxcare’s vendor customers and their end
users with Linuxcare’s services? Well, maybe Linuxcare has an edge
here, because their performance is typically being compared with
the notoriously bad tech support-related services that people are
used to receiving under the Cathedral model. But Linuxcare’s
ongoing surveys of end user satisfaction continually show that the
“people with the problems” are delighted with the quality
of the service they receive, including the speed with which they
receive it.
The large vendors who are paying Linuxcare to provide the
services are typically having their expectations about the value of
Linuxcare’s services to their businesses “dramatically exceeded.”
Linuxcare’s corporate customers are steadily expanding the scope of
work they are having Linuxcare do, and new corporate customers are
signing up. At the beginning of this year, several financial
analysts noted that Linuxcare’s customer base was still fairly
small and thus revenues could dip abruptly if any one of their
biggest customers were to cancel their contracts. Sifry, Tyde, and
LaDuke say they are not concerned about such a possibility
today — yes, the customer base has expanded, but the feedback they
are getting from their major customers is so positive that they
regard the possibility of a customer defection as “extremely
unlikely.”
If the August announcement of Linuxcare’s latest $30 million
round of funding is any guide, its corporate customers are
very happy indeed: Dell (through Dell Ventures), Motorola and Sun
Microsystems, major Linuxcare customers who contributed to earlier
rounds of funding, came back to contribute in this round. The
funding will keep Linuxcare on track in its move toward the IPO
milestone that Red Hat, Cobalt Networks, and VA Linux achieved in
’99. Other major customers that keep coming back for more Linuxcare
services include Compaq Computer Corp., Fujitsu Siemens Computers,
Hewlett-Packard, and IBM.
Isn’t the Distro Complexity a Nightmare to Deal With? —
No
People familiar with Cathedral-style tech support-related services,
where the focus is on the software products from a single vendor,
tend to shake their heads in disbelief when they first hear that
Linuxcare supports all major Linux distributions.
Basically, if there is enough demand for a particular distro in
business use, as long as the distro uses a standard Linux kernel,
Linuxcare will add it to the their knowledgebase. In addition,
Linuxcare supports the most common applications that run on the
different distros, such as Apache, KDE and GNOME, and StarOffice.
Isn’t this an absolute nightmare of complexities to deal with? —
Won’t they eventually be forced to pull back and concentrate on the
top five or so most popular “standard” Linux distributions?
Tyde’s answer: No. Why? “There’s no such thing as a ‘standard’
Linux distribution.” Remember that Linuxcare’s customers are mostly
large vendors. In practice, these customers typically take a
“standard” distribution like Red Hat and customize it to
meet their specific needs. Thus Linuxcare has to keep track of all
the details of each customer’s distro customizations anyway. After
gearing up to have the Linuxcare knowledgebase accommodate that
level of complexity, extending the scope to encompass the other
major distros didn’t really complicate things that much more.
Linuxcare does draw a line at kernel
customizations. All the information in the knowledgebase relates to
unmodified versions of the Linux kernel, as released by the kernel
team. They can provide services in support of software
that involves kernel modifications, such as Beowulf clustering, but
it’s not supported by the knowledgebase. It relies on the expertise
of individual technicians, tends to be extremely labor-intensive,
and hence tends to be more expensive to provide.
Some Examples of the Surprises
The Linuxcare founders had enough background in Cathedral-style
tech support-related services, and enough imagination to foresee
the Bazaar-style opportunities, so that they were able to
accurately anticipate the general outlines of the services that
their customers (large vendors) would want. What surprised them was
the timing.
It was anticipated that customers would initially sign up for
the basic tech support service — farming out tech support
to Linuxcare — and then gradually try out additional
services as they saw the benefits of the basic service and gained
confidence in Linuxcare. What they found was that customers were
eager to sign up for professional services (customization,
porting, integration, migration), product certification, and
training much earlier than they had expected.
Below are some examples of services that Linuxcare customers
requested fairly soon after they had signed up for “basic” tech
support. These are services that the original Linuxcare business
plan envisioned offering, just not so soon.
- Dell, one of Linuxcare’s biggest customers,
shipped a Linux workstation with an older version of a video card
driver that turned out to be buggy (it was the version that came
with Red Hat 6.0). Linuxcare detected the problem because several
Dell customers had called in. When you ran X windows, the clock
slowed down and ran at 1/8 speed, so it took 8 minutes to do 1
minute’s worth of work. After Linuxcare gave Dell the feedback on
the problem, Dell sent a workstation to Linuxcare, which identified
the problem in the video card and got Dell the correct driver. Dell
burned a new version of Red Hat with the updated driver to fix the
factory image. - Netaid/KPMG called Linuxcare because their SSL
performance was not fast enough. They were getting 10-15 tps
(transactions per second) per box, which was way too slow. They
needed a sustained 700-1000 tps, but were balking at having to pay
for 70 boxes to get it. Linuxcare tested three different solutions
and determined that Rainbow was the best one even though the Linux
crypto-card driver was buggy. Linuxcare rewrote the kernel driver
and Netaid/KPMG was able to get the SSL performance they needed by
purchasing only three servers, and one of those was just for
failover. - A major hardware vendor wanted to roll out
their new laptop. They came to Linuxcare to do a custom Linux build
because in the standard Caldera 2.4 build most notebook related
features didn’t work right — they couldn’t get full functionality
of some important features like power management, sound card, video
card, and networking capabilities. Linuxcare got Caldera Linux up
and running with full functionality. They even got the sound card
to suspend and resume better than it did on Windows, which the
hardware vendor said could not be done. On Windows it would suspend
and then give you a blue screen, which they said was due to a BIOS
bug, but Linuxcare went ahead and patched the Linux kernel to work
around it. The hardware vendor’s acceptance team was “blown away”
by the results that Linuxcare provided.
Linuxcare’s experience with ramping up their Bazaar business
model has validated — with a vengeance — the basic open source
concept about how money should be made with open source software.
The real value lies in services that make the free code do
exactly what people need it to do, within the specific
technical environment that they need it to do it. The availability
of such services is so revolutionary, in comparison with what
people are used to settling for with proprietary software, that
customers really are eager to pay what the services cost
— it’s “a bargain at twice the price!” For open source businesses,
like Linuxcare, that want to sell such services, the challenge is
gearing up to be able to provide all the services that
their customers will ask for.