Microsoft issued a statement on May 12 requesting the Linux
Community to respond to Mindcraft’s Open Benchmark challenge. In an
apparent attempt at “baiting” the Community, Microsoft digs deep
into the FUD barrel — a surefire way to win the confidence and
cooperation of the Community.
Concentrating on the Mindcraft study rather than the FUD, there
is a great deal to be learned from the PC
Week benchmark cited in the Microsoft statement. It appears
that the Linux Community’s concerns about the initial Mindcraft
test have largely been validated, namely, that the results of the
initial Mindcraft test did not accurately represent the performance
capabilities of Linux.
First, lets look at the WebBench numbers. PC Week attained up to
2,100 requests/sec with Apache on Linux. An amazing 110%
improvement over Mindcraft’s 1,000 requests/sec result.
Secondly, the NetBench numbers tell a similar story. Linux with
Samba achieved up to 197Mbps, a significant 73% improvement over
Mindcraft’s figure of 114Mbps. What’s more, drilling deeper into
the PC Week article, we find that the NT server achieved a meager
150Mbps of throughput when tested against WinNT clients. In others
words, the Samba/Linux combo outperformed Windows NT server by a
very healthy 31% when tested against Microsoft’s self-proclaimed
business class desktop product. PC Week goes on to state, “More
importantly, Samba had minimal performance degradation at higher
client loads. In tests with 60 clients, Windows NT managed only
110M-bps throughput compared with 183M-bps for Samba.” Strangely,
the Microsoft statement makes no mention of these results.
So what exactly is Microsoft complaining about? The PC Week
results make a strong case that Mindcraft did not properly tune
their Linux configuration. The Open Benchmark would likely yield
similar results. Microsoft’s demand that the Linux Community
“withdraw their criticisms of the initial Mindcraft report”, if
they do participate in the Open Benchmark, rings hollow. Moreover,
Mindcraft states that the purpose of the Open Benchmark is “to see
if Mindcraft’s second benchmark results are biased and not
representative of Linux’s true performance”. But what of the
initial Mindcraft report? Apparently, not even Mindcraft has
confidence in the numbers from the initial report.