What sites tell GPTBot and other AI crawlers

Archive of robots.txt and llms.txt policies. Look up one domain. If it is missing from the register it is added and scanned. HTML pages are not scraped.

No crawling of pages

Only HTTP GET against /robots.txt, /llms.txt and /llms-full.txt on the target host.

Bots we score

GPTBot, ClaudeBot, GrokBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and the rest of the dictionary — plus the robots.txt wildcard.

API

Public endpoints live under /v1/public/. History and compare need an API key. See the endpoint guide.