Linux Today: Linux News On Internet Time.

More on LinuxToday

SitePoint: Run Your Own Spider

Sep 30, 2004, 10:00 (0 Talkback[s])
(Other stories by Blane Warrene)


Desktop-as-a-Service Designed for Any Cloud ? Nutanix Frame

"I came across Carlos Perez's blog, manageability.org, while Googling for some research today. Carlos had a great list of open source web crawlers that included JSpider, a tool I have used for error checking on web sites.

"JSpider is written entirely in Java and can be configured extensively for spidering, error checking and downloading. It of course obeys robots.txt files and additional options included in configuration..."

Complete Story

Related Stories: