[ Thanks to Ken Sims
for this article. ]
Last Monday, Coverity, in collaboration with Stanford
University, announced
the results of their analysis of software quality and security
of 32 of the most critical and widely used open source projects in
the world. The study, which was funded by the Department of
Homeland Security, used Coverity’s automated defect detection tools
to uncover critical software bugs. In general, the analysis showed
that open source applications have lower defect rates than
proprietary software applications. The average defect rate of the
open source applications was 0.434 bugs per 1000 lines of code.
This compares with an average defect rate of 20 to 30
bugs per 1000 lines of code for commercial software, according
to Carnegie Mellon University’s CyLab Sustainable Computing
Consortium.
While this is a strong testament to the open source development
model, an even more interesting story occurred after the release of
the Coverity report. Of the 32 projects evaluated Amanda, an open
source backup and recovery project, had the highest number of bugs
per 1000 lines of code. The initial evaluation found a total of 108
bugs, or 1.22 bugs per 1000 lines of code. Clearly, this was of
concern to the Amanda community and those of us at my company
Zmanda (which is building a
business to provide enterprise support and services for Amanda
users.)
What happened next is truly remarkable. The Amanda development
community, which includes several Zmanda engineers, quickly
responded to address this situation. Within one week, Amanda
developers fixed the entire list of identified bugs. As it
currently stands, there are 0 outstanding bugs detected by the
Coverity scan and Amanda is
the most defect free open source project currently being evaluated
by Coverity.
Open source developers have immense pride in the quality of the
work.
Just as the chef in an open air kitchen knows that his cooking
will be viewed by all his restaurant patrons, an open source
developer is fully aware that his code will be scrutinized by
others. It will be subject to constant QA by developers, users and
analysis tools such as Coverity. This clearly results in higher
quality software. Perhaps even more powerful though, is the
capacity of a passionate open source community to deliver
astounding results when their work has been questioned. I am in awe
of what the Amanda community was able to accomplish. It’s unlike
anything that I’ve seen in more than 20 years in the commercial
enterprise.
Ken Sims is Vice President of Business Development and
Marketing at Zmanda, a
provider of Open Source Data Protection software and
services.