Linux Today: Linux News On Internet Time.

Mark Russinovich on Linux Versus Windows NT and Benchmark Reliability

Apr 30, 1999, 16:50 (75 Talkback[s])

Reprinted by permission from Windows NT Magazine UPDATE - Special Edition

by Mark Russinovich

Hello all,

Linux has received considerable attention from the trade press and major computer companies over the past several months. More and more, you see the word "enterprise" in a headline with Linux. For example, "The Enterprise Comes To Linux," "Enterprise Vendors Line Up Behind New Red Hat Linux Distribution," and "Major Players Bring Linux to Enterprise" are all recent headlines from popular trade rags. Major players in the computer world, such as IBM, HP, and Compaq, have announced formal support for Linux, and have begun selling hardware preinstalled with Linux. The ties between Linux and enterprise computing extend to software vendors who have announced Linux ports for applications common in the enterprise. Oracle has ported its database offering to Linux, IBM has ported DB2, and enterprise resource planning (ERP) solution providers are porting, or considering porting, their applications to Linux.

The promotional momentum behind Linux as an enterprise-ready OS gives the IS community a sense that Linux is a suitable enterprise OS that has passed the scrutiny of enterprise-savvy hardware and software companies. However, no evidence exists that these companies or the trade press has looked at Linux critically with respect to its enterprise-class capabilities. Enterprise-class environments don't widely use Linux--corporate environments have only just put Linux to use as a uniprocessor server OS in the past year or two. To date, not one Linux benchmark submission to the Standard Performance Evaluation Corporation (SPEC) or the Transaction Processing Council (TPC) has shown that Linux offers performance comparable with what commercial versions of UNIX and Windows NT deliver on large workloads. In fact, there are no SPEC or TPC benchmark submissions with Linux as the OS.

The phrase "enterprise computing" means different things to different people. Some see enterprise computing as a client/server environment where many clients, or users, rely on few servers to provide computing services. Thus, an enterprise environment is one that demands hardware resources to handle large workloads that basic PCs just can't support. My interpretation of the phrase defines an enterprise-class OS as one that must deliver high performance in high-load environments, scale well on multiprocessors, provide reliable service, and have high- availability options built-in or that exist as add-ons. Example enterprise environments include running an email server that supports several thousands of users, hosting a Web site that processes hundreds of thousands or millions of requests per day, and maintaining a database that serves as a corporate data repository with thousands or tens of thousands of employees.

A common theme to all these interpretations is that enterprise computing involves a network server application with high compute and I/O demands, and that server reliability and availability are of utmost importance to a company's bottom line. Another common theme is that the necessary hardware typically consists of a multiprocessor system with two to eight processors, 1GB or more of RAM, and dozens of GB of disk space: Uniprocessor departmental servers, small Web sites, and casually used company servers are not the enterprise. What lies beyond these requirements is the high-end of the enterprise (requiring more than eight processors). That space probably accounts for less than a fraction of 1 percent of the computing in the world and is reserved for mainframe OSs such as OS/390 or highly-clustered UNIX solutions.

An enterprise-level OS must meet several requirements. First, the OS must be reliable--it must not crash, even under extreme loads. Second, availability services must be in place to let clients continue to have their processing requests satisfied, even when the OS or hardware fails. This functionality implies redundancy at the hardware level, with OS or application support to redirect client requests from a failed system to one that is functional. Finally, the OS kernel's architecture and implementation must support high performance and scalability. In other words, the kernel must be sufficiently parallel in its operation to take advantage of multiple processors. The kernel must also provide applications with the APIs and infrastructure necessary to build high-performance, scalable applications.

How does Linux measure up to these requirements compared to the current players in the enterprise space (i.e., UNIX and NT)? Linux has a reputation as a reliable OS, and plenty of Web sites rely on it. However, its high-availability support is still virtually non-existent. Its Beowulf clustering, which has been in the trade press recently, is not suitable for providing high availability to network server applications. The only other measure to use to compare Linux to the rest of the field is performance, including multiprocessor scalability.

As I previously mentioned, no industry-accepted benchmark results exist for Linux running network server applications. Preparing for and executing such a benchmark is very costly, both in terms of hardware dollars and technical expertise required to execute a benchmark test that will yield competitive numbers. Thus, the lack of any Linux benchmark results means one of three things. First, no company with an interest in promoting Linux has run benchmarks. Second, it's not in the best interest of any company capable of submitting a benchmark with good Linux numbers to do so. Or third, companies are trying to get good Linux numbers and are failing.

It's not clear which companies capable of generating Linux benchmark results have it in their best interest to do so. Compaq, HP, Sun Microsystems, and IBM have all made pronounced gestures in support of Linux, and so they are all obvious candidates. However, they all have proprietary OSs that will compete with Linux as soon as someone demonstrates that Linux is ready for the enterprise. In fact, each company has its own version of UNIX. So it's not obvious which reason explains the lack of Linux numbers. The best we can do is analyze the Linux kernel to determine whether its design and implementation is capable of delivering good numbers.

Over the past few months, several writers in the technical press have thrown out claims regarding the capabilities of the Linux kernel. However, none of the authors that I've come across back their claims with any technical facts. As someone that specializes in OS internals and network server application design, I decided to examine the Linux kernel to determine whether it incorporates several key features required to implement high-performance network server applications. I discovered that Linux is lacking in several critical areas and that these limitations will hurt Linux's performance when compared to various versions of UNIX and NT. I published my findings in "Linux and the Enterprise," Windows NT Magazine, April 1999. (Windows NT Magazine has decided to post the electronic form of this article to its Web site at this time to help dispel some misplaced Linux hype and to open discourse on this subject. Go to http://www.winntmag.com/Magazine/Article.cfm?ArticleID=5048.) Although my current findings are negative, the intent of revealing them is not. I hope to promote Linux as a competitor in the enterprise area, as competition is good for the industry and good for consumers.

A report by the analyst group D. H. Brown entitled, "Linux: How Good Is It?" includes a study of the Linux kernel that comes to the same conclusion that I did. In fact, much of this group's conclusion related to Linux SMP support is based on my research--my preliminary findings during my study of the Linux kernel were posted, without my knowledge or permission, to the Linux kernel USENET newsgroup in January by a member of the Linux community. Pages 44 and 45, the heart of the SMP analysis, mirror my email almost exactly. The fact is, Linux has a kernel that internally looks very much like the prevalent UNIX kernel of the mid-1980s. All commercial versions of UNIX and NT have evolved 10 to 15 years past where Linux is as far as SMP scalability and network-server application support. And the gains that have been made for these other OSs have been difficult to achieve and expensive.

Despite a prevalent misconception in certain circles to the contrary, NT does classify as an enterprise-class OS. NT's performance on network server applications rivals or beats the performance the various versions of UNIX can provide. Don't take my word for it, go to http://www.specbench.org or http://www.tpc.org and see which OSs own the top three positions on systems varying in size from uniprocessor to 8-way systems. Across the board, NT ranks at the top or near the top when competing against the major UNIX variants, including Tru64 UNIX (formerly Digital UNIX), Solaris, and HP/UX. As far as the belief widely espoused in advocacy USENET newsgroups and Web sites to the contrary, NT is a reliable OS with high-availability options. If it wasn't reliable, no company would use it as an enterprise OS, and companies with no particular allegiance to Microsoft wouldn't use it at all. The number of companies visibly using NT in an enterprise-serving role are plenty: Walt Disney, Dell, Intel, eBay, Barnes and Nobles, United Airlines, Buy.com, and Car Point are just a sampling of some of the world's most demanding Internet sites running NT.

Will Linux gain the capabilities of its cousins and NT? Yes. Can it gain them in the next few years based solely on work done by a band of loose-knit developers scattered around the world? That remains to be seen. The lessons offered by the evolution of the other UNIX variants suggests that Linux requires a sponsor with experience in this area to help both financially and technically. Perhaps we'll see the formation of an industry consortium that will work together to help evolve Linux. However, it will be at least 2 years before the Linux kernel can mature to a competitive level.

As a closing note, I want to provide a warning concerning independent party OS benchmarks. Several publications have recently reported benchmark results comparing Linux to NT. Some have shown NT to outperform Linux, and some have shown the opposite. My advice is to disregard independent tests because they suffer from several flaws. First, by definition, an independent tester has no vested interest in the outcome of a benchmark. Thus, they also have no interest in understanding the hardware, OS, application, or benchmark used in ways required for achieving the best results possible. Only someone with a vested interest in good numbers can do the best job possible to achieve those numbers.

Second, independent tests are almost never performed using industry- accepted benchmarks such as SpecWeb or TPC-C/TPC-D. They are typically run using workload programs developed in-house or by third parties. Only tests that competing members of an industry deem representative of real-world environments can be readily accepted as meaningful.

Finally, no one audits independent tests. Representatives of each member of the benchmark consortium review benchmark results submitted to TPC or SPEC. This review ensures that these organizations will publish only numbers obtained in adherence with benchmark guidelines, and that means the submitter's competitors have critically viewed the numbers.


Mark Russinovich,
Contributing Editor, Windows NT Magazine

Copyright 1999, Windows NT Magazine