Debian Project Leader Says LSB Results Aren't Ready for Prime TimeOct 20, 2000, 17:49 (20 Talkback[s])
(Other stories by Michael Hall)
By Michael Hall, LinuxToday
In a message to the Debian developers list, Debian Project Leader Wichert Akkerman said early results from the Linux Standards Board's Filesystem Hierarchy Standard tests were misleading due to factors including incomplete installations on the test systems and bugs in the tests themselves.
According to Akkerman, though the Debian Woody installation tested failed 17 of the 243 items in the test for FHS compliance, only nine of the problems identified in the test were the result of actual non-compliance on the part of the distribution.
"Not all of the test results are fair in my opinion: some are real bugs in Debian, others are bugs in the testsuite or the result of using an incomplete install," said Akkerman in his message. He went on to comment on the prematurity of making claims about the quality of a distribution based on these results:
"Please also note that the testsuite isn't finished yet, and neither is the LSB standard, so it is too early to draw any conclusions as to how compliant any distribution will be to the final product. So do not be fooled by things like the SuSE press release, who seem to be ignoring that little fact."
Akkerman's comment was in reaction to yesterday's press release by SuSE, in which representatives of the company used the test results to claim their distribution is "the most standards-compliant Linux distribution tested."
In a followup letter to the list, Andrew Josey, LSB Test leader, and author of the LSB-FHS test suite, allowed that the test and the specification it was designed for still have some "issues:
"We should not be expecting any distributions to pass the current version of the test suite. Although we believe it to be a fair and accurate test of the LSB FHS 2.1 specification, there are issues with the specification and tests that need to be resolved. The policy with test development is that we test the specification "as is", and it's the specification owners that get to judge whether the spec is right or otherwise and not the test suite developers. Going forward if we were to roll out a formal certification process, and issues are agreed with the spec owners then we will modify the test accordingly or issue waivers."
Results of the distributions tested so far are listed below, and taken from The LSB's test results page.
Distribution Version # Tests Passed # Tests Failed -------------------------------------------------------------- Caldera 2.4 (Intel) 204 39 Debian Woody (Intel) 226 17 Kondara Server 2000 205 38 Mandrake 7.1 (Intel) 200 43 OpenLinux eServer 2.3 (Intel) 199 44 Red Hat Linux 7.0 (Intel) 223 20 Red Hat Linux 6.2 (Intel) 209 34 SuSE Linux 6.4 (Intel) 219 24 SuSE Linux 7.0 (Intel) 238 5 Turbo Linux 6.0.5 (Intel) 219 24