For which bot
User-agent determines which bot a group of rules applies to. Check the general group with an asterisk and the separate settings for the search engines you need.
Loading page
This may take a few seconds.
Loading page
This may take a few seconds.
If reChecker is not working correctly, describe the problem and the page where it happened. Leave an email or phone number; we will reply after checking.
Checking and parsing robots.txt
About the check
Robots.txt controls how search engine crawlers crawl a site. The analyzer parses rule groups, Allow and Disallow directives, and sitemap links, and flags issues in the file.
User-agent determines which bot a group of rules applies to. Check the general group with an asterisk and the separate settings for the search engines you need.
Disallow restricts crawling of specified paths, while Allow permits it for matching URLs. An empty Disallow value does not block the entire site.
The Sitemap directive specifies the full sitemap URL. Check that the link points to an existing file, not an old domain or an inaccessible page.
Practice example
This example explains Googlebot rules, not the results of checking your site. First, the User-agent group is selected, then URL paths are matched.
User-agent: *
Disallow: /catalog/
Allow: /catalog/public/In this group, /catalog/item/ is blocked from crawling, while /catalog/public/item/ is allowed: the Allow rule matches the path more specifically.
Google selects the most specific rule based on path length. If conflicting Allow and Disallow rules are equally specific, Allow takes precedence. Paths are case-sensitive: /Catalog/ and /catalog/ are different.
Disallow restricts crawling, but a URL may still appear in search results. Google must crawl the page to see noindex in a meta tag or X-Robots-Tag. Blocking it in robots.txt at the same time prevents this. Documentation about noindex ↗
After the check
First, match the issues to the purpose of the site sections. Crawl restrictions and indexing blocks serve different purposes.
Check directive names, colons, path values, and UTF-8 encoding. After making changes, open the published robots.txt and run the analysis again.
Make sure the rules aren't accidentally blocking important sections, CSS, or JavaScript. Blocking resources can prevent search crawlers from accessing page content.
A URL blocked in robots.txt may still appear in search results. To prevent indexing, use noindex on a page accessible to crawlers. Protect private data with authentication.
Robots.txt is a file in the site root with rules for search engine crawlers. It controls which pages are crawled, but doesn't guarantee they will be excluded from search results. A URL blocked by robots.txt may still be indexed through external links. Protect private data with authentication.
The public report shows the check date, overall score, errors found, and measured values.
We describe data sources, crawler limitations, counting rules, and cases that require a manual review.
We don't publish anonymous quotes or ratings without a verifiable original source. Case studies will be published only with the owners' consent.
Get help understanding the report, fixing technical issues or setting up regular monitoring. We will discuss the scope and cost before starting.