AI access gates
If a crawler can't get in, nothing else matters. We test each AI search bot separately.
- robots.txt rules for OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot and more
- CDN / WAF blocks and challenges (Cloudflare & co.)
- Content visible without JavaScript – most AI crawlers don't run it
- noindex, nosnippet and canonical traps