Red Hat Joins Debian Leader in Questioning Test Results, LSB Chair: Tests Are a “Work in Progress”

By Michael Hall,

Last week Debian Project Leader Wichert Akkerman fired off a
message to the Debian developer’s list taking issue with a press
release from SuSE claiming the German distribution is the “most
standards-compliant Linux distribution tested” based on the
strength of results from the LSB FHS test suite.

Red Hat’s Vice President of Engineering Erik Troan echoed
Akkerman’s concerns in a statement, saying that Red Hat will comply
with the LSB when it’s complete:

“As evidenced by Debian’s response to the test, this is a
standard that’s still under development. While we haven’t reviewed
the tests which are used in any detail, we are not surpised that we
have a similiar number of “failures” as other distributions. This
test suite looked for some items which are wrong (such as the
location of termcap and the name of the mtools configuration file),
some items which need further debate, (such as requiring the tclX
program /usr/bin/tcl), and some things which are quite new (such
as/etc/opt). When the LSB is a complete, implementable standard Red
Hat will follow it enthusastically. Comparing levels of conformance
to standards which are still undergoing active work is of
questionable value, however.”

According to Linux Standard Base Steering Committee Chairman Dan
Quinlan, Akkerman’s message “definitely raised some valid points,
all of which are being investigated for FHS 2.2.” In a separate
message to the Debian developer’s list, Quinlan addressed many of
the points in Akkerman’s mail.

Quinlan also acknowledged that the test suite is still

“The LSB-FHS test-suite is a work in progress, the bugs are
mostly due to problems in FHS primarily due to the evolution of
Linux. The first version of FHS (called FSSTND back then) was
released in 1994, but until recently, we didn’t have a test-suite.
Developers reading the FHS generally filter out minor issues so
they can go unreported for a while. Test suites developed using
rigorous methodology are not so forgiving.”

“Linux distributions have put a lot of effort into FHS
compliance and SuSE should be happy about their low number of
failures. Their results were very good. Unfortunately, the
announcement was premature because (1) it’s not really possible to
know how any distribution will fare when the LSB-FHS test suite is
finished, (2) it’s possible that there are other distributions that
will do just as well or better, (3) the test suite is still in
development, and (4) the test results posted are known to be
inaccurate and there are some bugs with FHS and the LSB-FHS test
suite (the LSB web pages said as much last week).”

Quinlan pointed out that there’s a new warning at the top of the
results page
reading “This is a work in progress, no claims of
LSB compliance or passing these test suites should be made. These
are unapproved test suites and issues remain both with the test
suites and the specifications under test.”

Get the Free Newsletter!

Subscribe to Developer Insider for top news, trends, & analysis