dcsimg
Linux Today: Linux News On Internet Time.




More on LinuxToday


SitePoint: Run Your Own Spider

Sep 30, 2004, 10:00 (0 Talkback[s])
(Other stories by Blane Warrene)

WEBINAR:
On-Demand

Re-Imagining Linux Platforms to Meet the Needs of Cloud Service Providers


"I came across Carlos Perez's blog, manageability.org, while Googling for some research today. Carlos had a great list of open source web crawlers that included JSpider, a tool I have used for error checking on web sites.

"JSpider is written entirely in Java and can be configured extensively for spidering, error checking and downloading. It of course obeys robots.txt files and additional options included in configuration..."

Complete Story

Related Stories: