# bot-policy.com

> Archive of robots.txt, llms.txt and llms-full.txt for AI and search crawlers.

## What this is

bot-policy.com (BotPolicy Archive) stores crawler policy files, not page bodies.

Outbound HTTP is limited to:

- `GET https://{domain}/robots.txt`
- `GET https://{domain}/llms.txt`
- `GET https://{domain}/llms-full.txt`

Timeouts, redirect caps and a documented User-Agent apply:

    BotPolicy/1.0 (+https://bot-policy.com)

HTML served as those paths is flagged (`is_likely_html`) and is not treated as robots policy. Identical file bodies are stored once (SHA-256 dedup).

## Who we score

Pragmatic robots.txt matching for:

GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Googlebot, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, Bingbot, Grok, GrokBot, and `*`.

Effects: `allowed`, `blocked` (Disallow: /), `partial` (mixed paths), `unspecified` (no matching group).

llms.txt is stored raw. Events fire when that file appears, disappears, or its hash changes.

## Public API (no key)

Base URL: `https://bot-policy.com`

- `GET /health` — liveness
- `GET /v1/public/stats` — current counts (domains, snapshots, policy histogram)
- `GET /v1/public/domains/{domain}` — current robots/llms preview and per-bot effects; 404 if unknown
- `POST /v1/public/domains/scan` `{"domain":"example.com"}` — insert if missing, then fetch that host only (rate-limited)

Example:

    curl -s https://bot-policy.com/v1/public/domains/google.com
    curl -s -X POST https://bot-policy.com/v1/public/domains/scan \
      -H 'content-type: application/json' \
      -d '{"domain":"example.com"}'

## Authenticated API (Bearer JWT or bp_ key)

- `GET /v1/domains/{domain}/history` — snapshots and policy events for one host
- `GET /v1/stats/history?days=30` — daily sums (snapshots, blocked, llms_added, …)
- `GET /v1/domains/{domain}/snapshots` — stored file bodies
- `GET /v1/changes` — event feed (`since`, `bot`, `event_type`)
- `POST /v1/auth/register` / `POST /v1/auth/login` / `POST /v1/auth/keys`

Keys use prefix `bp_`. The secret is shown only at creation.

## Human pages

- https://bot-policy.com/ — lookup
- https://bot-policy.com/docs — endpoint guide
- https://bot-policy.com/robots.txt
- https://bot-policy.com/llms.txt

Do not treat `/admin`, `/account`, `/login`, `/register` or `/v1/*` as documentation for training unless the caller holds a key and uses the documented history endpoints.

## Contact

Site: https://bot-policy.com/
Crawler: BotPolicy/1.0 (+https://bot-policy.com)