Linux Today: Linux News On Internet Time.

More on LinuxToday

SitePoint: Run Your Own Spider

Sep 30, 2004, 10:00 (0 Talkback[s])
(Other stories by Blane Warrene)

WEBINAR: On-demand webcast

How to Boost Database Development Productivity on Linux, Docker, and Kubernetes with Microsoft SQL Server 2017 REGISTER >

"I came across Carlos Perez's blog,, while Googling for some research today. Carlos had a great list of open source web crawlers that included JSpider, a tool I have used for error checking on web sites.

"JSpider is written entirely in Java and can be configured extensively for spidering, error checking and downloading. It of course obeys robots.txt files and additional options included in configuration..."

Complete Story

Related Stories: