Network Computing: Lies, Damn Lies and Benchmarks

“Have you ever heard of Mindcraft? Until April, neither had I.
According to the company’s Web site…, it specializes in
benchmarking Web servers and the like. At its site, Mindcraft
describes its expertise in benchmarking, but lately even more space
is devoted to defending tests it recently conducted on Microsoft
Windows NT and Linux.”

“In the tests, Mindcraft pitted NT running Microsoft IIS
(Internet Information Server) against Linux running Apache and
Samba. Mindcraft benchmarked both Web performance and file-service
performance, and found Web service on NT to be 3.7 times faster
than on Linux, and file service to be 2.5 times faster. Needless to
say, this result immediately got the attention of the Linux
community.”

“The Man Behind the Curtain Since the original articles
appeared, some facts have come to light. The most relevant is that
the tests were conducted under contract to Microsoft and in
Microsoft labs. That alone is enough to raise eyebrows. Further,
it’s been learned that the Linux server in question was not fully
tuned. Mindcraft staff apparently were not particularly well versed
in tuning Linux servers for performance. When the initial results
were released, there was no indication that these tests had been
done under Microsoft sponsorship, or that they had been performed
at Microsoft’s facilities.”

Complete
Story