Verifies a site's robots.txt syntax and provides a plain-English summary of the rules found in the file.

Provides the latest robots.txt rules from the Internet, so bots don't need to crawl publisher sites excessively.

Provides the latest robots.txt rules for a set of site urls. Rules come in two formats: raw or structured.

Continuously crawls and audits robots.txt for protection and updates as new bots emerge.