"At the time that robots.txt is being processed, there is no way
of telling which of these five cases will apply. For this reason,
robots.txt merely checks the ID, as it were, of the spider making
the request, for use by intelligence in subsequent requests.
"We can handle case number three by emitting stanzas that only
apply to the robot making the request. That way, there are no other
robots mentioned whose permissions it can record and later
"Now that robots.txt is actually a dynamic document, let's put
it to work and actually do something with it. For starters, and for
performance, I use a MySQL database to record bot activity and
access rules. Let's begin by having it record the particulars of
the current request. Here is the first MySQL table I use for
Some of the products that appear on this site are from companies from which QuinStreet receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. QuinStreet does not include all companies or all types of products available in the marketplace.