“HTML permits the inclusion of metadata in the header of the
document. This metadata is not intended for direct human
consumption, but is instead meant for programs to get additional
data about a Web page for automated indexing or other
collation. Because the metadata categories aren’t defined by
the HTML specification, the metadata is merely by convention. But
in particular, two kinds of metadata are understood by most of the
search engines: keywords and description…”
“But suppose you’ve gone to the trouble to make good
descriptions and keyword lists for all your major entry points. Why
wait for a spider to direct people there? Couldn’t you use the
information yourself, and build an index table to help people find
related pages on your site? Sure! Of course, you could do it all by
hand, but that might become time-consuming to update. So, let’s
write a program that spiders our own site, and generates a nice
index of all the keywords…”