What is crawling
Crawling is the process by which search engine bots (crawlers) automatically visit web pages. Googlebot, YandexBot, and other bots systematically follow links and save page content for later indexing.
Crawlability is the degree to which a website’s pages are accessible to crawlers.
The crawling and indexing process
1. Краулинг → 2. Парсинг → 3. Индексация → 4. Ранжирование
(обход) (анализ HTML) (добавление (позиции
в базу) в поиске)
A page must pass all four stages to appear in search results.
Causes of crawling problems
1. Blocked in robots.txt
# Запрет на обход всего сайта
User-agent: *
Disallow: /
# Запрет конкретных разделов
User-agent: Googlebot
Disallow: /admin/
Disallow: /private/
2. noindex directive
<!-- Страница не будет индексирована -->
<meta name="robots" content="noindex">3. Links hidden behind JavaScript
Links added by JavaScript after a page loads are less accessible to crawlers. Googlebot executes JavaScript, but with a delay.
4. Slow loading
Crawl budget is the crawling budget. Slow pages use it inefficiently.
5. Too many redirects
Long redirect chains slow down crawling.
6. Duplicate content
A large number of duplicates “dilutes” the crawl budget across irrelevant pages.
Crawl budget
Crawl budget is the number of pages Googlebot is prepared to crawl over a given period. It depends on the website’s authority and server speed.
For websites with thousands of pages, it is important to:
- Block unnecessary URLs (parameters, duplicates) using robots.txt or canonical
- Improve server speed (TTFB)
- Keep the sitemap updated with current URLs
How to improve crawling
robots.txt:
- Открыть важные разделы
- Закрыть дубли, технические страницы
sitemap.xml:
- Добавить все важные URL
- Актуальный lastmod
Структура сайта:
- Плоская иерархия (не глубже 3-4 кликов)
- Внутренние ссылки на важные страницы
- Нет orphan pages (страниц без ссылок)
Сервер:
- Быстрый TTFB (< 200 мс)
- HTTP 200 для существующих страниц
- HTTP 404 для удалённых
Check robots.txt on reChecker
Use the robots.txt analyzer to check your robots.txt file. The tool shows:
- The contents of robots.txt and whether it is accessible
- Blocked and allowed paths
- The sitemap link in robots.txt
- Directives for specific bots
FAQ
How can I find out which pages Googlebot has crawled? Google Search Console → Coverage → “Valid” and “Error.” You can also analyze server logs for the User-Agent: Googlebot.
Can I speed up crawling for a new website? Yes: submit a sitemap in Google Search Console, use the Google URL Inspection Tool for priority pages, and make sure robots.txt does not block important sections.
If a page is blocked in robots.txt, will Google index it? No. robots.txt Disallow prevents crawling—without crawling, there is no indexing. However, Google may learn about the URL from links on other websites and show an “empty” result without content.