Dynamic Robots.txt Rules for Apache Security

"This one is rather more complex, so let's go through it field by field, and follow that with some examples. The value of the 'pattern' field is key; it is to this value that aspects of the client request are compared to see if a particular rule matches or not. It might contain a string or a regular expression; how it is interpreted is controlled by the 'mode' field.

"Ordinarily, the value of the user-agent line of the robots.txt stanza will come directly from the request's 'User-agent' header field. The 'alias' field in the table provides a means to override this. For instance, the rule may actually have matched Firefox, but you can say that it matched Opera.

"The awkwardly-named 'field' field specifies which aspect of the request is to be matched against the pattern. I have found use only for the user-agent and the IP address, but there is no reason others might not be used."

Complete Story

Related Stories:

• Anatomy of a Web attack(Feb 05, 2009)
• 3 Apache/httpd Command-Line Options(Feb 03, 2009)
• Apache Basics, Visited(Jan 29, 2009)
• Apache Security: A Watched Bot Never Spoils (Your Server)(Jan 28, 2009)
• Installing Apache Tomcat on Linux(Jan 21, 2009)
• January 2009 Web Server Survey: Apache Shows Large Gains(Jan 19, 2009)