Parallel Processing Shell Script Released

[ Thanks to louwrentius for this link. ]

"There are many tutorials about how to execute commands in parallel, often using shell scripts. However, they are often not atomic, so parallel jobs may start processing the same file, causing trouble. Also, most examples on the web are based on some polling mechanism in order to determine when to start the next job. Polling is inefficient: you waste time between de moment a script finishes and the moment that event is picked up. Therefore PPSS is event-driven. It does absolutely nothing, thus saving CPU cylcles, while being as time-efficient as possible. The moment a job is finished, it will start a new one.

"The script will take a list of items as input. The script executes a user-specified command for each item in this list. The item is supplied as an argument to this command. Items can be files within a directory or entries in a text file (one per line)."

Complete Story

Related Stories:

• Speed Up Multiple SSH Connections to the Same Server(Dec 29, 2008)
• Basic Veritas Cluster Server Troubleshooting(Nov 16, 2008)
• Distributed Compilation: A Programmer's Delight(Nov 13, 2008)
• Scale Your File System With Parallel NFS(Nov 13, 2008)
• Parallel SSH Execution and a Single Shell to Control Them All(Oct 31, 2008)
• Supercharge Your LAN With Condor, part 1(Sep 18, 2008)