Response to Microsoft: PC Week benchmarks reveal Mindcraft failingsMay 14, 1999, 08:34 (78 Talkback[s])
Microsoft issued a statement on May 12 requesting the Linux Community to respond to Mindcraft's Open Benchmark challenge. In an apparent attempt at "baiting" the Community, Microsoft digs deep into the FUD barrel -- a surefire way to win the confidence and cooperation of the Community.
Concentrating on the Mindcraft study rather than the FUD, there is a great deal to be learned from the PC Week benchmark cited in the Microsoft statement. It appears that the Linux Community's concerns about the initial Mindcraft test have largely been validated, namely, that the results of the initial Mindcraft test did not accurately represent the performance capabilities of Linux.
First, lets look at the WebBench numbers. PC Week attained up to 2,100 requests/sec with Apache on Linux. An amazing 110% improvement over Mindcraft's 1,000 requests/sec result.
Secondly, the NetBench numbers tell a similar story. Linux with Samba achieved up to 197Mbps, a significant 73% improvement over Mindcraft's figure of 114Mbps. What's more, drilling deeper into the PC Week article, we find that the NT server achieved a meager 150Mbps of throughput when tested against WinNT clients. In others words, the Samba/Linux combo outperformed Windows NT server by a very healthy 31% when tested against Microsoft's self-proclaimed business class desktop product. PC Week goes on to state, "More importantly, Samba had minimal performance degradation at higher client loads. In tests with 60 clients, Windows NT managed only 110M-bps throughput compared with 183M-bps for Samba." Strangely, the Microsoft statement makes no mention of these results.
So what exactly is Microsoft complaining about? The PC Week results make a strong case that Mindcraft did not properly tune their Linux configuration. The Open Benchmark would likely yield similar results. Microsoft's demand that the Linux Community "withdraw their criticisms of the initial Mindcraft report", if they do participate in the Open Benchmark, rings hollow. Moreover, Mindcraft states that the purpose of the Open Benchmark is "to see if Mindcraft's second benchmark results are biased and not representative of Linux's true performance". But what of the initial Mindcraft report? Apparently, not even Mindcraft has confidence in the numbers from the initial report.