Free SEO & social tool
Robots.txt & Sitemap Validator
Check robots.txt line by line, see which search and AI crawlers may visit, and make sure your sitemap works.
- Free
- No sign-up
- Results in seconds
Full audit
This is one part of the full report
Speed, Core Web Vitals, UX, SEO and security in one free report, graded A+ to F, with AI fixes for your platform.
What it checks
robots.txt syntax
Typos, unknown directives and stray rules, by line number.
Search crawlers
Whether Googlebot, Bingbot and others can crawl your site.
AI crawlers
GPTBot, ClaudeBot, PerplexityBot, Google-Extended and 8 more.
URL tester
Test any path against any crawler, using Google’s matching rules.
Sitemap health
Valid XML, URL count, last-modified dates and host mismatches.
Sample URLs
A spot check that the first pages in your sitemap actually load.
Questions
What does robots.txt do?
It tells crawlers which parts of your site they may visit. It doesn’t hide pages from search by itself: a blocked page can still be indexed if other sites link to it. Use a noindex tag for that.
Should I block AI crawlers?
It’s your call. Blocking training bots like GPTBot and Google-Extended keeps your content out of future models. Blocking search bots like OAI-SearchBot also keeps you out of AI answers that could send you visitors.
How do you decide whether a URL is allowed?
The same way Google does. We find the most specific user-agent group for the crawler, then the longest rule that matches the path. When an Allow and a Disallow match equally, Allow wins.
How big can a sitemap be?
50,000 URLs or 50 MB per file. Bigger sites split their sitemap into parts and list them in a sitemap index.