Community: Open Scrutiny of Open Source CodeMar 18, 2006, 15:00 (12 Talkback[s])
(Other stories by Ken Sims)
[ Thanks to Ken Sims for this article. ]
Last Monday, Coverity, in collaboration with Stanford University, announced the results of their analysis of software quality and security of 32 of the most critical and widely used open source projects in the world. The study, which was funded by the Department of Homeland Security, used Coverity's automated defect detection tools to uncover critical software bugs. In general, the analysis showed that open source applications have lower defect rates than proprietary software applications. The average defect rate of the open source applications was 0.434 bugs per 1000 lines of code. This compares with an average defect rate of 20 to 30 bugs per 1000 lines of code for commercial software, according to Carnegie Mellon University's CyLab Sustainable Computing Consortium.
While this is a strong testament to the open source development model, an even more interesting story occurred after the release of the Coverity report. Of the 32 projects evaluated Amanda, an open source backup and recovery project, had the highest number of bugs per 1000 lines of code. The initial evaluation found a total of 108 bugs, or 1.22 bugs per 1000 lines of code. Clearly, this was of concern to the Amanda community and those of us at my company Zmanda (which is building a business to provide enterprise support and services for Amanda users.)
What happened next is truly remarkable. The Amanda development community, which includes several Zmanda engineers, quickly responded to address this situation. Within one week, Amanda developers fixed the entire list of identified bugs. As it currently stands, there are 0 outstanding bugs detected by the Coverity scan and Amanda is the most defect free open source project currently being evaluated by Coverity.
Open source developers have immense pride in the quality of the work.
Just as the chef in an open air kitchen knows that his cooking will be viewed by all his restaurant patrons, an open source developer is fully aware that his code will be scrutinized by others. It will be subject to constant QA by developers, users and analysis tools such as Coverity. This clearly results in higher quality software. Perhaps even more powerful though, is the capacity of a passionate open source community to deliver astounding results when their work has been questioned. I am in awe of what the Amanda community was able to accomplish. It's unlike anything that I've seen in more than 20 years in the commercial enterprise.
Ken Sims is Vice President of Business Development and Marketing at Zmanda, a provider of Open Source Data Protection software and services.