A customer clicks “Show more” and gets the next batch of products. A search crawler, however, may find only the cards in the first batch. A working interface confirms that loading works for a person, but it does not confirm that the rest of the catalog is accessible for crawling.
Check the path from the category to a specific product card. The next page should have a reproducible address, the transition should be an accessible link, and the product card should have a working URL. Neither the number of products in the admin panel nor a successful API response proves that this path exists.
Define what needs to be discovered
Start with a small category whose contents you can check manually. Save its address, sort order, batch size, product count, and several cards from the beginning, middle, and end of the list. Separately select a product that does not appear on the initial load. It will serve as the control point for the entire investigation.
Record the expected behavior without tying it to the current implementation: visitors and crawlers can reach every active product card through the catalog; directly loading the next page returns the correct part of the list; changing the page number does not unexpectedly change the category or filter.
In a hypothetical furniture catalog, the first batch might show tables, while the last shows chairs in the same category. Checking only the first card will not reveal that the last batch is empty because of an error in the query limit. So select products based on their position in the list and product type. General rules for navigation depth are covered in the article on site architecture and URL structure.
Find a transition that does not require a button click
Google explains that crawlers generally extract URLs from links and do not click buttons that require user action. Loading more results should be accompanied by an accessible route to those results. This is the key difference between an interface that works and navigation that works for crawling. Google’s pagination guidelines.
Open the category’s source HTML and find the link to the next page. Then check the DOM after the initial JavaScript execution, before clicking anything. Note whether it contains an actual address for the next batch or only an event handler. A regular button can remain for visitor convenience if there is also a route suitable for crawling.
For an accessible link, Google recommends an a element with an href attribute containing a resolvable address. Links added by JavaScript can also be processed if they use this markup. Therefore, the issue depends on the state of the link, not on JavaScript being used. Google’s documentation on accessible links.
Check the destination address separately: copy it and open it in a new tab without first visiting the category. If it works only after a token is obtained in the browser or a script runs on the previous page, that is a separate direct-access defect.
Compare direct loading with the loaded batch
There should be a clear correspondence between the button and the page at the direct URL. For one selected state, compare the product card IDs or URLs in the API response, the updated interface, and the page opened separately. If you compare only the category name, it is easy to mistake different sets of products for the same one.
Use this working protocol:
| Check | What to record | Sign of a problem |
|---|---|---|
| Initial load | URL and first group of cards | The category is present, but there are no links to products |
| Next batch | Number and URLs of the new cards | Products from the previous batch are repeated |
| Direct URL | Response code and list contents | The first page opens instead of the selected page |
| Last page | Last cards and navigation | The last products are not accessible through links |
| Page number beyond the list | Response and visible message | Empty pages keep appearing indefinitely |
Record the date of the check: the catalog changes, and the product sequence may differ tomorrow. A product card ID and query parameters are more useful for reproducing an issue than the statement “the product is missing from page two.”
Check the URLs and canonicals of list pages
Each page in a sequence has a different main set of products. So do not automatically treat them as duplicates of the first page. Google recommends separate URLs for list segments and a canonical URL pointing to the corresponding page, rather than one canonical URL for every segment pointing to the beginning. The URL section in the pagination documentation.
In your protocol, compare the actual URL, canonical URL, and “next” and “previous” links. If a direct load of page two declares page one as canonical, check the template configuration. Duplicate versions of the first page—for example, one with a parameter and one without—should also follow a consistent rule.
Do not automatically apply this solution to sorting and filters. They can create a different class of URL with a separate indexing policy. For this task, first restore the sequence of the main list. Combining a pagination fix with filter changes will increase the scope of the work and make acceptance testing more difficult.
If the canonical URL in the HTTP HTML differs from the canonical URL in the DOM, save both values. The difference indicates two configuration sources. The same approach to comparing representations is described in the article on JavaScript page content for search crawlers.
Separate product discovery from product indexing
The appearance of a link to a product means its URL can be discovered through this route. It does not mean that a search engine has already crawled and included the product page in its index. Do not close the task based on a search results screenshot: it does not show how the crawler obtained the URL.
Check the control product card: the final HTTP response, main text, indexing permissions, canonical URL, and presence of links from the list. If the route works but the product remains out of the index, continue the investigation at the page level and in the search console data.
A sitemap helps report product URLs, but it does not replace checking navigation. Customers still need links through the catalog. Record these as two separate observations: “the product is present in the sitemap” and “the product is reachable from the category.” This helps the team avoid confusing an additional discovery channel with a fix to the interface.
Define the task with a verifiable outcome
A practical task description: “For the category and every active list page, provide a direct URL and sequential links; control product cards from the last batches must be accessible without clicking the load-more button.” Attach a table of URLs and a snapshot of the current responses.
Set acceptance criteria before development:
- the next URL opens in a new browser session;
- the accessible markup contains a link to the next part of the list;
- the products on the direct page correspond to its page number;
- the canonical URL is consistent with the URL of the corresponding part;
- the last part is not lost, and nonexistent page numbers are handled according to the agreed rule.
When crawling is limited, a reChecker site audit can help check discovered URLs and technical signals. The number of discovered pages alone does not prove the catalog is complete: compare it with the prepared register of control products.
Accept the fix on a changing catalog
After release, repeat the same route and save the new results alongside the old ones. Then change the browser window size, clear the filters, and check a category with fewer products than one batch. This catches edge-case errors that are not visible on a long list.
Separately check what happens when a product is added or removed between loads. List shifting can sometimes cause a card to be repeated or skipped. Discuss this behavior with the developer as a property of the selected sorting and pagination mechanism; do not silently treat it as proof that all products are inaccessible.
The technical task is complete when the route can be reproduced, the URLs and list contents are consistent, and the control product cards are reachable. Crawling and indexing data should be checked later in the search console. Even correct pagination does not guarantee that every product card will be indexed or increase its ranking.