AI crawlers

GPTBot, ClaudeBot, PerplexityBot, and the rest of the curated index.

Recognised crawlers

Needle (UA)	Name	ai_source
`gptbot`	GPTBot	chatgpt
`oai-searchbot`	OAI-SearchBot	chatgpt
`chatgpt-user`	ChatGPT-User	chatgpt
`claudebot`	ClaudeBot	claude
`claude-user`	Claude-User	claude
`anthropic-ai`	Anthropic	claude
`perplexitybot`	PerplexityBot	perplexity
`perplexity-user`	Perplexity-User	perplexity
`google-extended`	Google-Extended	gemini
`googleother`	GoogleOther	gemini
`bytespider`	Bytespider	other
`applebot-extended`	Applebot-Extended	other
`ccbot`	CCBot	other
`amazonbot`	Amazonbot	other
`meta-externalagent`	Meta-ExternalAgent	other
`cohere-ai`	Cohere	other
`youbot`	YouBot	other

Public index

GET /api/crawler-index returns this list without auth. Use it to generate robots.txt allow-lists or to audit which bots you have actually seen via ai_traffic.

robots.txt

Vector 5 does not change your robots policy. It only measures who arrived. The Audit AI crawler access skill drafts an allow-list from observed traffic.