Linux Today: Linux News On Internet Time.
Search Linux Today
Linux News Sections:  Developer -  High Performance -  Infrastructure -  IT Management -  Security -  Storage -
Linux Today Navigation
LT Home
Contribute
Contribute
Link to Us
Linux Jobs


More on LinuxToday


SitePoint: Run Your Own Spider

Sep 30, 2004, 10:00 (0 Talkback[s])
(Other stories by Blane Warrene)

"I came across Carlos Perez's blog, manageability.org, while Googling for some research today. Carlos had a great list of open source web crawlers that included JSpider, a tool I have used for error checking on web sites.

"JSpider is written entirely in Java and can be configured extensively for spidering, error checking and downloading. It of course obeys robots.txt files and additional options included in configuration..."

Complete Story

Related Stories: