“Web client programming is a powerful technique for querying the
Web. A web client is any program that retrieves data from a web
server using the Hyper Text Transfer Protocol (the http in your
URLs). A web browser is a client; so are web crawlers, programs
that traverse the Web automatically to gather information. You can
also use web clients to take advantage of services offered by
others on the Web and add dynamic features to your own web
site.”
“Web client programming belongs in any developer’s toolbox.
Perl aficionados have employed it for years. In Python, the process
reaches even higher levels of convenience and flexibility.
Three modules provide most of the functionality you will need:
HTTPLIB, URLLIB, and a newer addition, XMLRPCLIB. In true
Pythonesque fashion, each module builds upon its predecessor,
providing a solid, well-designed base for your applications. We
will cover the first two modules in this article, saving XMLRPCLIB
for a later time.”
“For our examples, we will use Meerkat. If you are like me, you
invest time tracking trends and developments in the open source
community to give you a competitive edge. Meerkat is a tool that
makes that task much easier. It is an open wire service that
collects and collates an enormous amount of information on open
source computing. Although its browser interface is flexible and
customizable, using web client programming we can scan, extract,
and even store this information off-line for later use. We will
first access Meerkat using HTTPLIB interactively, and then move on
to accessing Meerkat’s Open API via URLLIB to create a customizable
information-collecting tool.”