Secure Apache: Out, Damned Bot

“Spiders and Flies

“The tools and robots that crawl the Web looking for content
(for whatever reason) are frequently called ‘spiders,’ or sometimes
‘bots.’ Some spiders are good, such as the Google bot, which loads
the Google search engine with what it finds. Others have a much
more questionable goodness quotient, such as those that search Web
pages for e-mail addresses to add to spam lists, or look for
trademark references so that the information can be sold to the
trademark holders for possible lawsuits.

“While the term spider is in common use, I’ve never heard anyone
give a name to the other type of abuse — that of hijacking
writable Web pages such as blog comments and wikis. I’m going to
coin the term ‘flies’ for abusive tools of this type, since they
cluster around and crawl all over pages, leaving flyspecks and crap
on them.”

Complete Story