The page opens in a browser, but its stylesheet and main JavaScript file are blocked from the crawler. Sometimes only a widget disappears without the script; sometimes the entire catalog fails to appear. First, identify which resources are actually needed to understand the page.
Which files generate text and product cards
Google warns that blocking necessary resources makes it harder to understand pages that depend on them. Official documentation.
The style.css resource may only control a button’s color, while app.js may be responsible for displaying the entire catalog. These resources being unavailable has different effects. Start by identifying the main content: the service name, description, products, and navigation links. Note which elements are already in the server-rendered HTML and which are created later. If the name, specifications, and purchase link are already in the HTML, losing a small widget is not the same as losing a product card. If the HTML is empty and the products are created by app.js, access to that file is critical. Allow necessary resources after this check.
Record the URLs of the resources you need
| Resource | What it is used for | Where to look for a restriction | What should work |
|---|---|---|---|
| /assets/app.js | Renders the catalog | Check the rule | Allow the required JavaScript file |
| /assets/style.css | Controls the display | Compare the rendering | Allow the required stylesheets |
| CDN on another host | Its own robots.txt | Check separately | Do not apply the domain’s rule to it |
| /internal/ | Internal section | Assess its purpose | Do not allow everything |
Compare the source HTML with the rendered page
Save the text and links from the source HTML, then compare them with the HTML after scripts have run — the DOM. In Search Console, use URL inspection and review the HTML and page screenshot if available. It is important to see more than just a screenshot: product cards may look normal but contain no actual links. A Console error may also explain why content is missing, independently of robots.txt. If a resource is blocked, check whether the relevant crawler group is actually disallowed from accessing its path. A successful load in your browser shows that the server works for a user, but browsers do not apply robots.txt the way search crawlers do.
Test JS dependency on a staging copy
On a staging copy, choose a page where a JavaScript file creates product cards. In Chrome developer tools — DevTools — open Network, right-click the script request, and select Block request URL. Reload the page and compare the products with the original state. Remove the block in the Network request blocking panel and reload again without changing the products. This test shows the content’s dependency on the resource, but it does not simulate the entire search processing pipeline. Afterward, check the actual robots.txt rules and server access. If the absence of this file does not change the main text, note that observation; do not automatically describe every JavaScript block as a critical loss of content.
Check the CDN robots.txt and renamed files
A CDN — a content delivery network — has its own domain, robots.txt, and responses. A rule on the main domain does not automatically control cdn.example.com. After a new build, the filename may change from app.abc.js to app.def.js; allowing only the old file will no longer help. It is better to allow the required folder if it contains only public resources, and check the new filenames after the build. Make sure switching to a CDN has not started loading HTTP resources on an HTTPS page (mixed content) or required authentication. Testing the new build should include fetching its actual JavaScript file, not a file left in the cache from the previous release.
For a resource directory, an example rule looks like this:
User-agent: Googlebot
Disallow: /assets/
Allow: /assets/public/This rule is suitable if the required scripts are actually located in /assets/public/. Check the full URL in Network and the existing Googlebot groups: do not add the example without considering the rest of the file.
Repeat the check with a fresh build
The required CSS/JS return 200 and are allowed for the selected crawler. Check the main text and links in the available HTML after the page has been processed in Search Console. If only a regular browser is available, note this: a successful load for you does not confirm that the search engine received the same content.
Read also: SSR vs CSR vs SSG for SEO: what to choose in 2026 and Robots.txt: a complete setup guide for SEO and web development. Website checks are available on the reChecker technical audit page.