Linux Today: Linux News On Internet Time.

Secure Apache: Out, Damned Bot

Dec 05, 2008, 19:33 (0 Talkback[s])
(Other stories by Ken Coar)

"Spiders and Flies

"The tools and robots that crawl the Web looking for content (for whatever reason) are frequently called 'spiders,' or sometimes 'bots.' Some spiders are good, such as the Google bot, which loads the Google search engine with what it finds. Others have a much more questionable goodness quotient, such as those that search Web pages for e-mail addresses to add to spam lists, or look for trademark references so that the information can be sold to the trademark holders for possible lawsuits.

"While the term spider is in common use, I've never heard anyone give a name to the other type of abuse -- that of hijacking writable Web pages such as blog comments and wikis. I'm going to coin the term 'flies' for abusive tools of this type, since they cluster around and crawl all over pages, leaving flyspecks and crap on them."

Complete Story

Related Stories: