The general User-agent: * block allows the directory, while a separate Googlebot group blocks it. User-agent selects a group of rules for a crawler, Allow permits the specified path, and Disallow prohibits it from being crawled. Checking only the general group gives the wrong answer. You also cannot apply one search crawler’s interpretation to another without checking the documentation and the specific set of rules.
Find each crawler’s dedicated group
Google selects the most specific matching User-agent group; the order of the groups does not determine priority. Official documentation.
Suppose the general block allows /catalog/, while the dedicated Googlebot group contains Disallow: /catalog/. In this case, reading only User-agent: * creates a false expectation. Put the groups in a table, preserving their exact lines. Separately note rules for specialized crawlers, if present. Googlebot-Image crawls images, and its restrictions do not mean the same rules apply to regular Googlebot. In the verification record, specify the crawler, domain, and full path: otherwise, results from different groups are easy to mix up.
User-agent: *
Allow: /catalog/
User-agent: Googlebot
Disallow: /catalog/
Allow: /catalog/public/For Googlebot, /catalog/a/ is blocked, while /catalog/public/a/ is allowed by a more specific rule. The general group does not supplement the dedicated group. For Yandex, check the actual file in its tool and robots.txt help.
Use the same test paths
| Group or rule | Applies to | Where to check the path | What to consider |
|---|---|---|---|
| User-agent: * | General group | Check a regular path | Do not treat it as the only group |
| User-agent: Googlebot | Dedicated group | Check the directory | Consider group selection |
| User-agent: YandexBot | Different policy | Check in Webmaster | Do not apply Google’s conclusion |
| Allow and Disallow | Overlapping paths | Compare specificity | Check the specific URL |
How Google selects a rule
According to Google’s documentation, group selection depends on the most specific matching User-agent, while selection of the matching path depends on rule specificity. Do not assume that the last line always wins. When rules overlap and have equal specificity, Google takes the Allow rule into account. Check this with a specific test URL, not by looking at the visual order of blocks in the file. If an agent is listed in multiple matching groups, check how the rules are combined in the current Google documentation. The result of this check applies to Google’s interpretation and is not a universal instruction for every internet bot.
Check the same paths for Yandex
In Yandex Webmaster, open the robots.txt analysis tool, paste in the current file, and check the same /catalog/a/ and /catalog/public/a/. Compare the result with the User-agent help and Allow/Disallow help. If a matching dedicated group exists, the general block * does not replace it. Save the result for YandexBot separately. If it differs from the intended result, fix the specific rule rather than deleting all groups. Without access to the analyzer, this check cannot yet be considered complete.
What a request with a different User-Agent shows
A request with the User-Agent: Googlebot header tests how the server responds to that string. It does not make curl or a browser apply robots.txt, and it does not confirm that the request came from a real search crawler. Treat the server-access test and the crawling-rules test as separate checks. If a security filter (WAF) distinguishes between agents, that is another separate policy to diagnose using logs. Do not present a successful 200 from a spoofed request as a full check of search crawling. Real requests require the intended crawler-authentication check and available logs.
Save the results of both checks
For /catalog/a/ and /catalog/public/a/, record separate results for Googlebot and YandexBot. After making a change, repeat the public retrieval of the file and check how these paths are interpreted. The neighboring restricted section should remain blocked by the selected rule.
Read also: robots.txt examples for different CMSs: WordPress, Bitrix, Joomla and 7 errors in robots.txt that kill indexing. Website checks are available on the reChecker technical audit page.