Parallel Processing Shell Script Released
Jan 06, 2009, 01:34 (1 Talkback[s])
Full Text Search: The Key to Better Natural Language Queries for NoSQL in Node.js
[ Thanks to louwrentius for this link.
"There are many tutorials about how to execute commands
in parallel, often using shell scripts. However, they are often not
atomic, so parallel jobs may start processing the same file,
causing trouble. Also, most examples on the web are based on some
polling mechanism in order to determine when to start the next job.
Polling is inefficient: you waste time between de moment a script
finishes and the moment that event is picked up. Therefore PPSS is
event-driven. It does absolutely nothing, thus saving CPU cylcles,
while being as time-efficient as possible. The moment a job is
finished, it will start a new one.
"The script will take a list of items as input. The script
executes a user-specified command for each item in this list. The
item is supplied as an argument to this command. Items can be files
within a directory or entries in a text file (one per line)."