Robots.txt Feed



Robots.txt Feed provides the most recent updated robots.txt rules from sites around the Web, all stored in our database. This feed is provided here to help publishers decrease bot traffic. Bots can access this feed instead of crawling your site repeatedly to check which rules have changed.

Feed Download

Generates a streamed JSON or NDJSON dump for a given bot and rules format.





Feed API Endpoint

Endpoint:GET https://robots-txt.ai/api/robots-txt-feed
Query parameters:bot: required bot name, maximum 20 characters
format: raw or structured (default: raw)
output: json or ndjson (default: json)
download: set to true to download the response as a file

Example with raw format (default)

Request:
GET https://robots-txt.ai/api/robots-txt-feed?bot=SomeBot&format=raw

Response:
[
  {
    "url": "https://example.com",
    "modified": "2026-08-08T13:40:50Z",
    "rules": "User-agent: SomeBot\nAllow: /\nDisallow: /private\nSitemap: https://example.com/sitemap.xml\nCrawl-delay: 5"
  }
]
Example with structured format

Request:
GET https://robots-txt.ai/api/robots-txt-feed?bot=SomeBot&format=structured

Response:
[
  {
    "url": "https://example.com",
    "modified": "2026-08-08T13:40:50Z",
    "rules": {
      "user-agent": "SomeBot",
      "allow": [
        "/"
      ],
      "disallow": [
        "/private"
      ],
      "sitemap": [
        "https://example.com/sitemap.xml"
      ],
      "crawl-delay": 5
    }
  }
]