Linux Today: Linux News On Internet Time.

Debian Project Leader Says LSB Results Aren't Ready for Prime Time

Oct 20, 2000, 17:49 (20 Talkback[s])
(Other stories by Michael Hall)

By Michael Hall, LinuxToday

In a message to the Debian developers list, Debian Project Leader Wichert Akkerman said early results from the Linux Standards Board's Filesystem Hierarchy Standard tests were misleading due to factors including incomplete installations on the test systems and bugs in the tests themselves.

According to Akkerman, though the Debian Woody installation tested failed 17 of the 243 items in the test for FHS compliance, only nine of the problems identified in the test were the result of actual non-compliance on the part of the distribution.

"Not all of the test results are fair in my opinion: some are real bugs in Debian, others are bugs in the testsuite or the result of using an incomplete install," said Akkerman in his message. He went on to comment on the prematurity of making claims about the quality of a distribution based on these results:

"Please also note that the testsuite isn't finished yet, and neither is the LSB standard, so it is too early to draw any conclusions as to how compliant any distribution will be to the final product. So do not be fooled by things like the SuSE press release, who seem to be ignoring that little fact."

Akkerman's comment was in reaction to yesterday's press release by SuSE, in which representatives of the company used the test results to claim their distribution is "the most standards-compliant Linux distribution tested."

In a followup letter to the list, Andrew Josey, LSB Test leader, and author of the LSB-FHS test suite, allowed that the test and the specification it was designed for still have some "issues:

"We should not be expecting any distributions to pass the current version of the test suite. Although we believe it to be a fair and accurate test of the LSB FHS 2.1 specification, there are issues with the specification and tests that need to be resolved. The policy with test development is that we test the specification "as is", and it's the specification owners that get to judge whether the spec is right or otherwise and not the test suite developers. Going forward if we were to roll out a formal certification process, and issues are agreed with the spec owners then we will modify the test accordingly or issue waivers."

Results of the distributions tested so far are listed below, and taken from The LSB's test results page.

Distribution    Version         # Tests Passed  # Tests Failed
Caldera         2.4 (Intel)          204             39 
Debian          Woody (Intel)        226             17 
Kondara         Server 2000          205             38 
Mandrake        7.1 (Intel)          200             43 
OpenLinux       eServer 2.3 (Intel)  199             44 
Red Hat         Linux 7.0 (Intel)    223             20 
Red Hat         Linux 6.2 (Intel)    209             34 
SuSE Linux      6.4 (Intel)          219             24
SuSE Linux      7.0 (Intel)          238             5 
Turbo Linux     6.0.5 (Intel)        219             24 

Related Stories: