Robots.txt Search



Robots.txt Search provides the latest robots.txt rules from sites around the Web, so publisher aren't overwhelmed with too many bot requests. Get the rules using our simple API described below. Check for updates every day; the maximum number of URLs per request is 1000. Set format to structured for parsed fields or raw for the relevant robots.txt content. Both formats return the result in the rules field. If omitted, it defaults to raw.


Endpoint: POST: https://robots-txt.ai/api/robots-txt-search
Headers:Content-Type: application/json
Accept: application/json
Accept-Encoding: br, gzip (optional): requests a compressed response when supported


Example with raw format (default)

Request:
{
  "urls": [
    "https://example-1.com",
    "https://example-2.com"
  ],
  "user-agent": "SomeBot",
  "format": "raw"
}
Response:
[
  {
    "url": "https://example-1.com",
    "modified": "2026-08-08T13:40:50Z",
    "rules": "User-agent: SomeBot\nAllow: /\nDisallow: /products\nDisallow: /private\nSitemap: https://example-1.com/sitemap1.xml\nSitemap: https://example-1.com/sitemap2.xml"
  },
  {
    "url": "https://example-2.com",
    "rules": "User-agent: SomeBot\nAllow: /news\nAllow: /blog\nDisallow: /\nCrawl-delay: 5"
  }
]

Example with structured format

Request:
{
  "urls": [
    "https://example-1.com",
    "https://example-2.com"
  ],
  "user-agent": "SomeBot",
  "format": "structured"
}

Response:
[
  {
    "url": "https://example-1.com",
    "modified": "2026-08-08T13:40:50Z",
    "rules": {
      "user-agent": "SomeBot",
      "allow": [
        "/"
      ],
      "disallow": [
        "/products",
        "/private"
      ],
      "sitemap": [
        "https://example-1.com/sitemap1.xml",
        "https://example-1.com/sitemap2.xml"
      ]
    }
  },
  {
    "url": "https://example-2.com",
    "rules": {
      "user-agent": "SomeBot",
      "allow": [
        "/news",
        "/blog"
      ],
      "disallow": [
        "/"
      ],
      "crawl-delay": 5
    }
  }
]