In response to criticism of its
Linux versus Windows NT benchmark comparisons commissioned by
Microsoft, Mindcraft is
withholding the publication of its second Linux and Windows NT
Server benchmark results pending an Open Benchmark
Invitation.
Mindcraft believes that the best way to continue to
show our name as a credible source of information is to have an
Open Benchmark. Therefore, we welcome the opportunity to perform a
benchmark of Linux and Windows NT Server that is open to the best
experts in the Linux community. Mindcraft will participate in this
benchmark at its own expense.
Linux Today applauds this invitation by Mindcraft. Free Software
and Linux stand much more to gain than they could ever lose in a
truly fair and objective comparison between Linux and Windows NT.
In fact, it is quite likely that any deficiencies in the software
would be quickly addressed by a swarm of Free Software
developers.
But Samba development team member, Jeremy Allison, takes issue
with one benchmark requirement stipulated by Mindcraft, in
particular that:
… the tests will use Windows 9x clients (these
can be in any mix including all clients using the same Windows 9x
OS). The same client set up will be used for both Linux and Windows
NT Server.
In a letter to Mindcraft’s Bruce Weiner, Jeremy stipulates some
requirements of his own:
In order to participate, I must insist that Windows
NT clients are *also* benchmarked.I feel that it is very much in the interest of consumers to
be able to choose between clients, and surely Microsoft would be
interested in publishing the results of their flagship workstation
operating system, running against their flagship server, using
their flagship filesystem.I insist that we also have NT clients running against NT
server and Samba servers, using an NTFS filesystem on NT, and an
equivalent secure filesystem (ext2fs) on Linux added to the
mix.There are no problems running Win9x clients against NT using
whatever file system you wish as well, but not also showing the NT
client data is *extremely* disingenuous, as I’m sure you already
know.Your original benchmark was funded by, and on behalf of
Microsoft, run in their labs (as you have now revealed). This new
benchmark is different and is a joint effort between Microsoft and
the Open Source community to publish meaningful numbers to allow
consumer choice. In order to be able to use our names on this we
must have some input into the benchmarking specifications.If you refuse to consider this important addition to the
matrix I will have no option but to publicly refuse to participate
and publish our own Win9x client to NT or Samba server, and WinNT
client to NT or Samba server benchmark matrix, and let our own
credibility be assessed by the public.
Linux Today endorses the view of Jeremy Allison that a truly
‘Open’ benchmark must allow both sides an equal opportunity to
stipulate testing conditions — particularly in this case where the
Allison proposal does not restrict the Mindcraft benchmark platform
in any way but merely adds an additional platform against which to
test.
The Jeremy Allison proposal should, in fairness to truth, be
accepted and then… let the benchmark proceed!