Robots.txt opens, but the Sitemap line points to an old domain or returns a regular HTML page instead of an XML sitemap. Fixing the line itself helps only if the destination file actually contains the pages you need. Open the specified URL, check the XML, and inspect a few pages listed in it. The word Sitemap appearing in robots.txt does not, by itself, confirm that the sitemap works.
Find all Sitemap lines
Google supports the Sitemap directive, which specifies the full URL of a sitemap. Official documentation.
A valid example line for illustration: Sitemap: https://example.com/sitemap.xml. The value must contain the full address, not a local path on disk or /sitemap.xml without a host. If the sitemap is named sitemap_index.xml, specify that exact URL. A robots file can contain multiple Sitemap lines, so find all of them, not just the first. Explain the purpose of each one. An old file or a sitemap for another project does not become appropriate just because its name looks familiar and the link technically opens.
Address and sitemap response errors
| Found in robots or in the response | Why it causes a problem | What to fix | Expected response |
|---|---|---|---|
| Sitemap: /sitemap.xml | Relative value | Replace with the full URL | Check the actual address |
| Old domain | Historical path | Update the destination | Check ownership and response |
| 200 with HTML | A template instead of a sitemap | Fix the route | Valid XML is needed |
| Sitemap index | List of files | Open the child files | Check each sitemap |
Check the file’s status and contents
For the sitemap URL, record the HTTP status, the Content-Type header—the declared file format—and the response body. HTTP 200 may accompany a login page, a web application firewall (WAF) placeholder, or a custom error page. Ask a developer to parse the returned XML, for example using the commands in the sitemap file check: it must have a valid root element, urlset or sitemapindex, in the specified format. For 301 or 302, check the entire redirect chain; a link to an intermediate path makes the configuration harder to understand. For 403, find out what is blocking public access without disabling the site’s protection entirely. If there is a timeout, check the generator and response size first rather than blindly resubmitting the sitemap.
Use a terminal command to download the sitemap; if you do not work with curl, ask a developer.
curl -sS -D headers.txt -o response.html 'https://example.com/sitemap.xml'Check the first line of headers.txt and the contents of response.html: the response should be the sitemap, not a site template. If you get a redirect, open the destination address from the Location response header separately. The output file extension does not determine its format—check the XML inside. Replace example.com with your domain; this GET request does not change the site’s settings. curl documentation.
The sitemap body starts with XML and contains urlset with pages or sitemapindex with links to sitemaps. A login page or HTML error at 200 means the required sitemap was not retrieved.
Open the section sitemaps
If the root element is sitemapindex, the file is a list of sitemaps. Its loc elements contain file addresses and point to files, not directly to pages. Request each child sitemap and check it separately. A main index with valid XML does not prove that products.xml is accessible and contains the required products. For compressed files, check that they can be decompressed and inspect their contents; for removed files, restore them or remove their links. Then open a sample of the page URLs in loc from each sitemap. For example, an accessible index will not help a catalog if products.xml returns an error. After checking the file, open several product URLs from its loc elements.
Change the link at its actual source
First, restore the sitemap itself at the required URL. Then change the Sitemap setting in the content management system (CMS) or project file that generates robots.txt. If a plugin generates robots.txt, a manual edit to the static response may disappear with the next update. Remove outdated lines only after confirming that their sitemaps are no longer needed. Keep a list of the required sitemaps so a new section is not accidentally left out. After publishing, repeat the GET requests for robots.txt and each destination: a new setting in the admin interface does not confirm that the public route is already serving it to visitors and crawlers.
Check robots and the sitemap at their regular URLs
After making the change, retrieve robots.txt and every listed sitemap again. The selected pages should be included in the appropriate sitemaps. If a sitemap is intentionally hosted on another site, check the cross-site sitemap requirements and ownership verification; a different host does not, by itself, prove there is an error.
Also read: Sitemap.xml errors: how to find and fix them and Robots.txt: a complete guide to configuration for SEO and web development. Site checks are available on the reChecker technical audit page.