The Outercite crawler

OuterciteBot/1.0 (change-verification; +https://outercite.com/crawler)

What it does

Change verification. When a customer tells us they changed a page, we fetch that one page, once, to confirm the change is live. One URL per verification run, no link following.

Weekly site read. Once every 7days, for each account that has verified a site, we read that site’s pages, starting from its sitemap and following links within the site, to check what the site itself says about the business: whether its pages can be found, and whether they state an address, a phone number and opening hours. At most 2,000 pages per run, one request at a time, at least 1 second apart, robots.txt and redirects included. It identifies itself as OuterciteBot/1.0 (site-read; +https://outercite.com/crawler).

What it will not do

  • Fetch a domain its owner has not verified. A typed URL is not permission; verification requires proving control of the domain (Cloudflare zone or DNS record).
  • Ignore robots.txt. A Disallow for OuterciteBot is honoured. A robots.txt we cannot read means we do not fetch: permission that cannot be checked is not assumed.
  • Follow a redirect off your site.A redirect from http to https, or to your own host’s www twin, is followed, after checking that address’s robots.txt. A redirect to any other host ends the fetch without being requested.
  • Leave your site. The site read stays on your own host and its www twin. Every redirect is checked before it is followed, and one that points at another domain is refused without being requested.
  • Store your pages. Verification keeps a content hash, a byte count, an HTTP status, and at most 200 characters of context per failing check. The site read keeps, per page, a content hash, its length and status, its canonical and robots directives, the structured data it publishes, the business details it states, and at most 200 characters of visible text. Never the page.
  • Fetch more than 1.5 MB for a page, or 10 MB for a sitemap file. Larger responses are abandoned mid-stream.

Compatibility probes

Separately, our free site check can test whether a site blocks a named AI crawler. Those probes send that crawler’s User-Agent token with our own identity appended, (compatibility probe by Outercite; +https://outercite.com), so your access log always shows who was actually asking.

Opting out

Add User-agent: OuterciteBot / Disallow: / to your robots.txt, or email team@outercite.com to opt a domain out entirely.

Machine-readable version: /.well-known/outercite-crawler.json