Linux Today: Linux News On Internet Time.

Ranting About Parallel I/O: We Need Standards!

Mar 24, 2011, 02:07 (0 Talkback[s])
(Other stories by Henry Newman)

"Search engines such as Google, open source Hadoop, databases such as Oracle and DB2, and many others are parallel applications.

"Parallel applications can do I/O in two different ways: They can write to a parallel file system such as GPFS, Lustre or Pan-FS, or they can write to a local file system on each server node in the configuration. The problem is that there is a dirty little secret: We have no standards other than the Message Passing Interface (MPI) for parallel processing communications and MPI-IO for parallel I/O. And I do not see any changes in sight because, in my opinion, the vendors that now control the standards bodies do not want to change anything as they can sell proprietary solutions that cost more in the long run, and lock you into that vendor. And it costs much more if you want to change vendors."

Complete Story

Related Stories: