The generator added /api/products/ alongside product pages. The response contains JSON—the app’s data—so it loads successfully, but it is not a page for visitors. Find the public product page that visitors should open, and keep its URL in the sitemap. Exclude the utility API URL that the app uses to get product data; a response of 200 does not make it a product page.
Identify the purpose of each URL
A sitemap helps notify a search engine about selected URLs, but does not guarantee that they will be indexed. Official documentation.
A content management system (CMS) may store /products/a/, /api/products/a, and /downloads/a.pdf in the same table. The first URL is a product page, the second is an app data source, and the third is a manual. Start with this classification rather than applying a universal filter by file extension. Each route should have a clear user and an expected response. If a URL is needed only for internal app communication, its appearance in the sitemap usually indicates that the wrong source was selected. JSON can be a standalone public document, so first find out who the URL is for. In this example, the API serves the catalog, while users need an HTML product page.
Product page, API, and PDF manual
| URL in the sitemap | Response format | What to select instead | What to check on the page |
|---|---|---|---|
| /product/a/ | HTML page | Keep it for its intended purpose | Content and canonical |
| /api/products/a | JSON data | Exclude the utility URL | Do not substitute it for the product page |
| /api/account | Personal information | Check authentication | A sitemap is not protection |
| /manual.pdf | Public document | Decide separately | The format itself is not a prohibition |
Compare Content-Type and the response body
Save the Content-Type and the start of the body for each suspicious loc. For JSON, check that it contains data, not an HTML error page incorrectly named as an API. For HTML, open the public-facing interface and compare its purpose. Check whether the utility URL redirects to a login screen. In some systems, URLs have no file extension, so a path like /data/a does not tell you the format. You should know where the URLs are generated: from app routes, CMS entries, or crawl results. If the URLs come from the CMS, change its content type filter. If they are collected by a crawler, find the page that links to the API as if it were a regular product page.
You can get the format of a utility URL with a terminal command; pass it to a developer.
curl -sS -D headers.txt -o response.html 'https://example.com/api/products/a'The headers in headers.txt show the Content-Type, while response.html shows the actual body of the URL from the sitemap. JSON with product fields does not become an HTML page because of status 200. Find and check the separate public product page URL. Replace example.com with your domain; this GET request does not change the site’s settings. curl help.
Content-Type: application/json and JSON containing product data indicate a data source. Find the actual HTML product page that visitors should reach. Removing the API from the sitemap does not block direct requests to it; personal data is protected with authentication.
Test the list with sample entries
Ask a developer to feed the generator three entries on a test copy: a public product page, a utility URL, and a published manual. Write down what you expect for each before running it. After generating the sitemap, check the actual loc entries. Then add a private URL and a draft document: they should not appear just because they have registered routes. Also check a utility URL without the /api/ prefix. If it appears in the sitemap, the generator excludes only a familiar path name rather than all utility entries.
Include only the public types you need
In the generator settings, list the public types you need, such as products and articles. Do not export all app routes: they may include form handlers, payment system callbacks, and admin panels. For a CMS, check the publication status and SEO exclusions for the relevant content type. If the generator gets URLs from a crawl, find out why public pages link to the API as a navigation destination. In that case, cleaning up the sitemap alone is not enough: you need to fix the source of those links. Keep the app API access required by real user-facing features.
Check the sitemap and catalog after the change
The product page remains in the sitemap, and its API is no longer listed. Open the product page directly and through the catalog: its data and photos should load. Keep or exclude the public PDF according to its purpose, not just its format.
Read also: Sitemap.xml: a complete guide to creating and optimizing it for SEO and HTTP status codes: a complete reference (200, 301, 404, 500...). Site checks are available on the reChecker technical audit page.