by Alan Cox
“Mindcraft did conduct a second test with support from..”
They (or rather he as it appears to be) _refused_ us access to
the system. I have basically given up on Mindcraft’s Microsoft
funded pranks.
The careful use of the word “support” to imply we somehow
validate his test is misleading. I sent him about four emails.
Only on May 4th did he offer any kind of open testing. The
response I sent him was very simple
> I think I would prefer to see an open benchmark done on
your
> configuration and test plan by an alternative testing body.
That
> would also be better for all parties.
I think I speak for much of the community in feeling that. No
such test has yet occurred. I have no faith that Mindcraft can or
would deliver an accurate assesment of the general question of
Linux v NT performance.
File Server Performance
———————–
I would also invite Mindcraft to publish the Windows NT client
numbers using NTfs. That is the equivalent benchmark to using Linux
where you also have file permissions and multiple timestamping on
files. Perhaps Mr Weiner has conveniently forgotten that Windows 98
is supposed to be the end of the line and Windows 2000/NT5, an NT
based product is the future according to the MS roadmap.
Bruce says that “Mindcraft verified the clients were set up as
we documented”. I do not believe anyone can verify an OS client was
set up as documented by a company engineer. This comment implies
that Mindcraft did not set up the client machines themselves. I
would hope they followed basic test quality considerations and set
the client machines up using install disks bought at a random
computer shop and that no third party engineer was able to access
the test site except under supervision. Who did set the client
machines up ? Because Bruce refused us access we don’t know.
Bruce Weiner says that Jeremy reported he duplicated the NT
server numbers in their lab. Jeremy also could not reproduce the
abysmal numbers Mindcraft got from their Linux box.
“The only reason to use Windows NT clients is to give
Linux and Samba an advantage”
Using NT clients is a more accurate measurement. Using NTfs
would also be more accurate. The file systems then are equivalent
in feature sets. It is also rather easier to tune a FAT file system
to put the files and blocks in perfect order for a test. I’d thus
want to be sure that the files were unpacked with a standard open
source tool on a newly formatted partition on each machine so we
can be sure nobody used any kind of fs optimiser on either the NT
or Linux box.
I could equally say
“The only reason to use Windows 98 clients its to give
NT an advantage”.
In fact since Microsoft supply both products in this case it is
a much more reasonable statement.
So why don’t they include NTfs benchmarks. I can only guess that
Bruce’s statement answers that in full, as well as telling a whole
story
“…why the company that sponsored a comparative benchmark
always came out on top. The answer is simple. When that was not the
case our client exercised a clause in the contract allowing them to
refuse use the right to publish the results. We’ve had several such
cases”
Flaws In Test Analysis
———————-
If Mr Weiner wants to get accurate OS assesments for for web
performance he should run Zeus. Last time I checked Zeus was the
fastest Linux web server, and IIS seems to be the fastest NT web
server. Linux + Zeus also appeared to be cheaper than NT Server
(including IIS). He acknowledges that Apache is not the fastest web
server for Linux, yet he did not choose to use the obvious product
for the testing.
He claims
“We showed that Windows NT server was faster than
Linux”
He states
“None of the Unix benchmark results reported at SPEC
use Apache”
Why then did he use Apache. He may be measuring Apache v IIS
speed but he is not measuring Linux v NT speed, and he knows
it.
He claims
“We showed that NT server was faster than
Linux”
He states
“Mindcraft used a MegaRAID controller with a beta
driver… PC Week server used an eXtremeRAID controller with a
fully released driver. The MegaRAID driver was single threaded
while the eXtremeRAID driver was multi-threaded”
By his own admission he is benchmarking an incomplete test
driver and claiming that it proves NT server is faster than
Linux.
I would like to see an honest truely open benchmark using the
best tools for each platform and showing the areas where each one
wins. I am sure there are some things NT is good for (the cynical
response of BSOD aside). Only an open benchmark by a good
responsible and trusted third party can provide the information
users need and have a right to know when selecting the right OS
product for a job.
End users have a right to honest open benchmarks they can trust
when selecting a product. It is sad that the prospect of an open
choice seems to scare some large vendors.