Robots & Sitemap Tester
Validate robots.txt and your XML sitemap
Reads your robots.txt, breaks out the rules per crawler, follows any declared sitemaps and validates the XML. The failure this catches most often is the one that matters most: a "Disallow: /" left in place after a launch, quietly keeping the whole site out of search results.
Frequently asked questions
Do I need a robots.txt?
Not strictly — without one crawlers assume everything is allowed. It is worth having anyway, if only to declare your sitemap and keep crawlers out of endless URL spaces like search and filter pages.
Does Disallow keep a page out of Google?
No. It stops the page being crawled, not indexed — a blocked URL can still appear in results from links alone. To keep a page out of the index use a noindex meta tag, and leave it crawlable so the tag can be seen.
Where should my sitemap live?
Anywhere, as long as robots.txt declares it with a Sitemap: line. /sitemap.xml is the convention and the location crawlers try first.