Free tool
Can AI crawlers read your site?
Enter a domain. We check robots.txt for the ten AI crawlers that decide whether ChatGPT, Claude, Perplexity, Gemini and Apple Intelligence can read and cite you, plus noindex, llms.txt and your sitemap.
About the check
What does the AI crawler checker test?
It reads your robots.txt and tells you, for ten AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot and Applebot-Extended), whether each may read your homepage. It also checks your homepage for noindex, how much text is visible without JavaScript, whether you have an llms.txt file, and whether your sitemap is reachable.
Should I allow every AI crawler?
Not necessarily. Search and user-request crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot) decide whether AI answers can cite you. Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) are a business choice. The checker shows what each one does so you can decide.
Do you store my data?
The check itself stores nothing about you. If you ask for the full report by email, we keep only that email address and the domain you checked.