Ranting About Parallel I/O: We Need Standards!

“Search engines such as Google, open source Hadoop, databases
such as Oracle and DB2, and many others are parallel
applications.

“Parallel applications can do I/O in two different ways: They
can write to a parallel file system such as GPFS, Lustre or Pan-FS,
or they can write to a local file system on each server node in the
configuration. The problem is that there is a dirty little secret:
We have no standards other than the Message Passing Interface (MPI)
for parallel processing communications and MPI-IO for parallel I/O.
And I do not see any changes in sight because, in my opinion, the
vendors that now control the standards bodies do not want to change
anything as they can sell proprietary solutions that cost more in
the long run, and lock you into that vendor. And it costs much more
if you want to change vendors.”

Complete Story