The robots.txt file is one of the fundamental elements of technical SEO optimization, but it is often underestimated by developers and website owners. This text file controls crawling: which pages search robots can request and which they cannot. This article covers how to configure it and avoid common mistakes.
What is robots.txt and why is it needed?
Robots.txt is a text file located in a website's root directory. It contains instructions for search robots and other web crawlers. The file is part of the Robots Exclusion Protocol (REP), developed in 1994 to give webmasters control over how automated programs interact with their websites.
The main purpose of robots.txt is to control search robots' crawling of website sections. Disallowing crawling does not guarantee that a URL will be removed from search results: the address may be known through external links. To exclude an accessible page from the index, use noindex, which the robot must be able to read. Protect private data with authorization and server settings.
History and development of the standard
The robots.txt standard was created by Martijn Koster in 1994. Initially, it was a way to tell search robots which parts of a website they should not crawl. Additional capabilities were introduced later, including the ability to specify sitemap.xml.
Directive support depends on the robot. Google understands User-agent, Allow, Disallow and Sitemap, but does not support Crawl-delay. Robots.txt rules do not control HTTP caching. Check the documentation for the relevant search engine to learn about additional directives.
Robots.txt structure and syntax
Basic syntax
The robots.txt file has a simple but strict structure. Each directive must be on a separate line, and the file must be located in the website's root directory (for example, https://example.com/robots.txt).
User-agent: *
Disallow: /admin/
Disallow: /temp/
Allow: /public/
Sitemap: https://example.com/sitemap.xmlUser-agent directives
The User-agent directive specifies which search robots the rules apply to. An asterisk (*) means the rules apply to all robots.
# Правила для всех роботов
User-agent: *
Disallow: /private/
# Специальные правила для Googlebot
User-agent: Googlebot
Disallow: /private/
Disallow: /googlebot-excluded/
# Правила для Bingbot
User-agent: Bingbot
Disallow: /private/
Disallow: /admin/In Google, a specific group User-agent is applied instead of the general group *, so any required general disallow rules must be repeated. The folder name /googlebot-excluded/ does not by itself set noindex.
Disallow and Allow directives
The Disallow directive prevents robots from accessing specified sections, while Allow permits access to specific subsections within disallowed areas.
User-agent: *
# Запрещаем доступ к административной панели
Disallow: /admin/
# Но разрешаем доступ к публичной части админки
Allow: /admin/public/
# Запрещаем доступ к временным файлам
Disallow: /temp/
Disallow: /cache/
Disallow: /tmp/
# Запрещаем доступ к файлам определенных типов
Disallow: /*.pdf$
Disallow: /*.doc$
Disallow: /*.xls$Sitemap directive
The Sitemap directive specifies the location of the sitemap.xml file, helping search robots find and index website pages more quickly.
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-news.xml
Sitemap: https://example.com/sitemap-products.xmlConfiguring robots.txt for different CMS platforms
WordPress
WordPress is one of the most popular CMS platforms, and configuring robots.txt correctly is essential for SEO.
The examples below are starting points, not universal files for every CMS installation. Do not block public CSS, JavaScript, or image files: robots need them to render the page. Robots.txt does not replace server-side protection for restricted files.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /xmlrpc.php
Disallow: /readme.html
Disallow: /license.txt
Disallow: /wp-config.php
Disallow: /wp-config-sample.php
Disallow: /wp-login.php
Disallow: /wp-register.php
# Разрешаем доступ к важным файлам
Allow: /wp-content/uploads/
# Указываем sitemap
Sitemap: https://example.com/sitemap_index.xmlJoomla
Joomla has its own specific requirements, and robots.txt for this CMS should account for its structure.
User-agent: *
Disallow: /administrator/
Disallow: /cli/
Disallow: /installation/
Disallow: /logs/
Disallow: /tmp/
Disallow: /configuration.php
Disallow: /htaccess.txt
Disallow: /web.config.txt
Disallow: /LICENSE.txt
Disallow: /README.txt
# Разрешаем доступ к изображениям и медиафайлам
Allow: /images/
Allow: /media/
Sitemap: https://example.com/sitemap.xmlDrupal
Drupal has a modular architecture, which is reflected in the structure of its robots.txt file.
User-agent: *
Disallow: /admin/
Disallow: /CHANGELOG.txt
Disallow: /COPYRIGHT.txt
Disallow: /INSTALL.mysql.txt
Disallow: /INSTALL.pgsql.txt
Disallow: /INSTALL.sqlite.txt
Disallow: /INSTALL.txt
Disallow: /LICENSE.txt
Disallow: /MAINTAINERS.txt
Disallow: /README.txt
Disallow: /UPGRADE.txt
Disallow: /xmlrpc.php
# Разрешаем доступ к публичным файлам
Allow: /sites/default/files/
Sitemap: https://example.com/sitemap.xmlOpenCart
For OpenCart online stores, it is important to configure robots.txt correctly to optimize product indexing.
User-agent: *
Disallow: /admin/
Disallow: /catalog/controller/
Disallow: /catalog/language/
Disallow: /catalog/model/
Disallow: /config/
Disallow: /download/
Disallow: /image/catalog/demo/
Disallow: /install/
Disallow: /logs/
Disallow: /storage/
Disallow: /system/
Disallow: /vendor/
# Разрешаем доступ к изображениям товаров
Allow: /image/catalog/
Allow: /catalog/view/javascript/
Allow: /catalog/view/theme/*/stylesheet/
# Ограничиваем обход ненужных вариантов фильтрации и сортировки
Disallow: /*route=product/search*
Disallow: /*sort=*
Disallow: /*order=*
Disallow: /*limit=*
Disallow: /*filter_*
Sitemap: https://example.com/sitemap.xmlAdvanced robots.txt configuration techniques
Separate rules for robots
A separate group lets you define rules for a specific robot. It does not control page caching and does not replace HTTP cache headers.
User-agent: *
Disallow: /admin/
# Дополнительный запрет обхода для Googlebot
User-agent: Googlebot
Disallow: /admin/
Disallow: /googlebot-excluded/
# Дополнительный запрет обхода для Bingbot
User-agent: Bingbot
Disallow: /admin/
Disallow: /bingbot-excluded/Google ignores Crawl-delay. Check the support and meaning of additional directives for other robots separately. The syntax and group selection are described in Google's official specification.
Blocking by file type
You can restrict crawling of certain file types. This does not prevent them from being executed on the server or guarantee that they will be excluded from the index. Google's rules support the * and $ patterns, but not regular expressions with \. or (pdf|doc).
User-agent: *
# Блокируем документы
Disallow: /*.pdf$
Disallow: /*.doc$
Disallow: /*.docx$
Disallow: /*.xls$
Disallow: /*.xlsx$
Disallow: /*.ppt$
Disallow: /*.pptx$
# Блокируем архивы
Disallow: /*.zip$
Disallow: /*.rar$
Disallow: /*.7z$
# Блокируем исполняемые файлы
Disallow: /*.exe$
Disallow: /*.msi$
Disallow: /*.dmg$
# Блокируем конфигурационные файлы
Disallow: /*.ini$
Disallow: /*.conf$
Disallow: /*.config$Managing URL parameters
One useful technique is blocking pages with certain parameters that can create duplicate content.
User-agent: *
# Блокируем страницы с параметрами сортировки
Disallow: /*?sort=*
Disallow: /*?order=*
Disallow: /*?filter=*
# Блокируем страницы с параметрами сессий
Disallow: /*?session=*
Disallow: /*?sid=*
Disallow: /*?PHPSESSID=*
# Блокируем страницы с параметрами отслеживания
Disallow: /*?utm_source=*
Disallow: /*?utm_medium=*
Disallow: /*?utm_campaign=*Do not automatically block paginated pages: robots may use them to find products and articles. Keep links accessible on every page you want crawled, and give each one its own canonical. See Google's pagination guidelines. The patterns above account for the parameter appearing after ?; check actual URLs, including the order of parameters.
Common robots.txt mistakes
Mistake 1: Incorrect file placement
Robots.txt must be located in the website's root directory, not in subfolders.
# Неправильно
https://example.com/blog/robots.txt
# Правильно
https://example.com/robots.txtMistake 2: Incorrect syntax
A common mistake is using incorrect syntax or omitting colons.
# Неправильно
User-agent *
Disallow /admin/
# Правильно
User-agent: *
Disallow: /admin/Mistake 3: Blocking important content
Developers often accidentally block access to important website sections.
# Неправильно - блокируем весь контент
User-agent: *
Disallow: /
# Правильно - блокируем только административные разделы
User-agent: *
Disallow: /admin/
Disallow: /private/Mistake 4: Missing sitemap
Many people forget to specify the location of sitemap.xml in robots.txt.
# Добавляем sitemap
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xmlTesting and validating robots.txt
Using Google Search Console
In Google Search Console, the robots.txt report shows the version of the file Google retrieved, any errors, and the date it was crawled. Use the URL Inspection tool to check whether a specific URL is accessible.
- Sign in to Google Search Console
- Select your website
- Open the robots.txt report and check the file retrieval status
- For an important page, run an inspection of the published URL
- Check whether crawling of the page is allowed
Using online tools
There are many online tools for testing robots.txt:
- reChecker robots.txt analyzer
- Bing Webmaster Tools
- Yandex Webmaster
- Specialized SEO tools
Manual testing
You can check robots.txt manually by accessing the file directly:
https://example.com/robots.txtMonitoring and analytics
Tracking crawling
Robots.txt does not prevent users from opening a page in a browser. Client-side JavaScript does not show which URLs a search robot requested; server logs are needed for that.
Analyzing server logs
Analyzing server logs helps you understand how search robots interact with a website.
# Поиск запросов от поисковых роботов в логах
grep -i "googlebot\|bingbot\|yandex" /var/log/nginx/access.log
# Анализ частоты запросов
awk '/googlebot/ {print $1}' /var/log/nginx/access.log | sort | uniq -cConclusion
Robots.txt controls how search robots crawl a website. Check that important pages and resources are accessible. To exclude a page from the index, use an appropriate method, such as noindex while allowing crawling; for private data, use server-side access controls.
Robots.txt rules are followed by robots that support the standard. It is not a security measure: an unwanted bot may ignore the rules.
After changing your website structure, check the rules again for the relevant pages, CSS, JavaScript, and images. Changing robots.txt alone does not guarantee improved rankings.
🔗 Check your robots.txt: rechecker.ru/services/robots-analyzer