Linux Today: Linux News On Internet Time.

SitePoint: Run Your Own Spider

Sep 30, 2004, 10:00 (0 Talkback[s])
(Other stories by Blane Warrene)

"I came across Carlos Perez's blog, manageability.org, while Googling for some research today. Carlos had a great list of open source web crawlers that included JSpider, a tool I have used for error checking on web sites.

"JSpider is written entirely in Java and can be configured extensively for spidering, error checking and downloading. It of course obeys robots.txt files and additional options included in configuration..."

Complete Story

Related Stories: