What is robots.txt
Robots.txt is a text file located in the website’s root directory (https://example.com/robots.txt), containing instructions for search engine crawlers about which pages they can and cannot crawl.
The file follows the Robots Exclusion Protocol (REP), adopted in 1994. Today, all major search engines support it: Google, Yandex, Bing, and DuckDuckGo.
It’s important to understand that robots.txt is a recommendation, not a command. Well-behaved bots follow it, but malicious actors may ignore it.
Why robots.txt is needed
A correctly configured robots.txt serves several purposes:
- Conserving the crawl budget — avoid spending the crawl limit on technical pages
- Protecting private sections — keep admin panels and API endpoints from appearing in Google
- Preventing duplication — block parameterized URLs (
?sort=,?page=) - Speeding up indexing — crawlers can focus on important pages
How to configure robots.txt
Basic file structure:
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/
Sitemap: https://example.com/sitemap.xml
Main directives
User-agent — specifies which crawler a rule applies to:
User-agent: * # все роботы
User-agent: Googlebot # только Google
User-agent: YandexBot # только Яндекс
Disallow — prohibits crawling:
Disallow: / # закрыть весь сайт
Disallow: /admin/ # закрыть раздел admin
Disallow: /*.pdf$ # закрыть все PDF файлы
Allow — allows exceptions to a prohibition:
User-agent: *
Disallow: /private/
Allow: /private/public-page # разрешить одну страницу
Crawl-delay — the delay between requests (supported by Yandex, but not Google):
User-agent: YandexBot
Crawl-delay: 1
Sitemap — a link to the sitemap:
Sitemap: https://example.com/sitemap.xml
Common mistakes
- Blocking CSS and JS files — prevents Googlebot from rendering pages
- Blocking /api/ — if these URLs are needed for the website to work
- Using
Disallow: /to hide content instead of noindex - Typos in paths —
/Admin/and/admin/are different paths
Check robots.txt with reChecker
Use the robots.txt analyzer to check your file. The tool will show:
- Syntax errors in the file
- Important pages that are blocked
- Whether a Sitemap link is present
- Warnings about potential issues
FAQ
Do I need robots.txt if there’s nothing to hide?
Yes. Even an empty robots.txt is useful — it lets you specify a Sitemap and shows crawlers that the file exists. Minimal configuration: User-agent: *, Allow: /, Sitemap: URL.
Does robots.txt hide pages from Google? No. Robots.txt prohibits crawling, but Google may still index a URL if other websites link to it. To exclude a page from the index completely, use meta robots noindex.
How often does Google reread robots.txt? Google caches robots.txt for 24 hours. After making changes, you can speed up the update through Google Search Console → URL Inspection → Request indexing.