A page opens for the site owner, but the search crawler receives a protective placeholder. Or the text is accessible, but an AI answer cites an outdated company directory. These are different problems: the first concerns content delivery; the second, source selection and accuracy. You can’t fix both with a single “AI file.”
Here, an AI audit means checking whether a website is ready to be read and used as a source. This is a practical guide, not a specialized reChecker service or an audit performed by a neural network. The outcome is a set of checked URLs, confirmed obstacles, sources for facts, and a plan for rechecking.
Operator rules were checked on 9 October 2026. Text availability is necessary for it to be retrieved, but does not guarantee indexing, a mention, or a citation.
Separate the three tasks before you begin
| Task | What to check | Where to find evidence |
|---|---|---|
| Technical page readiness | The bot receives the required text and applicable directives | HTTP, HTML, rendering, server logs |
| Visibility monitoring | A name, link, or quote appears in a specific answer | Saved query, mode, date, and answer |
| AI-assisted audit | A model helps analyze the data provided | Verify its conclusions against the source facts |
A brand’s absence from one answer does not prove that the website is blocked. Conversely, a successful request with a particular User-Agent does not mean that a service will recommend the company. For answer monitoring, see the separate guide to GEO and visibility.
Step 1. Choose pages containing the relevant facts
Start with the home page, a service or product page, terms, pricing, contact information, and one substantive guide. For each page, record the exact URL, its purpose, and the fact a user should be able to verify: service area, what the service includes, current price, or a product limitation.
Don’t include private client information to improve visibility. Keep authentication in place as a security measure. If some information can be published, prepare a separate, approved description and verify who owns it.
Create a table with these columns: URL, fact to check, HTTP, source text, DOM, robots, canonical, actual bot access, confirmation in an answer, action. For unknown fields, write “not checked,” not “no issues.”
Step 2. Get a response without a user session
Open the page in a private window. Then save the headers, final URL, and GET response body. Check the content itself: 200 with a login form, empty template, or CAPTCHA does not confirm that the page was delivered.
curl -sS -L -D /tmp/ai-audit-headers.txt \
-o /tmp/ai-audit-page.html https://example.com/serviceThe command creates local files; it does not execute JavaScript. Compare the final address with the expected page, and check Content-Type and X-Robots-Tag. To investigate periodic 403, 429, and 5xx, you’ll need repeated observations and logs from around the time of the failure.
A manual request with a bot’s User-Agent is useful for finding differences in responses, but it confirms only how that request was handled. The actual bot connects from a different IP; the CDN may apply a different rule to it.
Step 3. Check robots, noindex, and snippet restrictions
Collect the applicable robots.txt groups for the path and operator in question. Then check meta robots, bot-specific directives, and X-Robots-Tag in the final response.
| Mechanism | What it means for the audit |
|---|---|
Disallow in robots.txt | Restricts crawling of the specified path for the relevant bot |
noindex | Tells search engines to exclude the document from search; the bot must be able to retrieve the directive |
nosnippet | In Google, prohibits a text snippet and use of the content as a direct input to AI Overviews/AI Mode |
max-snippet | In Google, limits the amount of text available for previews and direct use in these AI features |
data-nosnippet | Excludes the marked passage from a snippet under Google’s rules |
These are Google’s rules, not a promise that all AI systems behave the same way. The basis is the robots meta and X-Robots-Tag documentation. For another system, check its terms.
A crawl restriction can prevent a bot from seeing noindex. If the page is meant to be public and indexable, fix the source of the unintended directive: the CMS, template, server, or CDN. If the restriction is intentional, keep it. Robots.txt does not protect data from visitors and does not replace password-protected access.
For Google Search AI features, the page must be indexed and eligible to appear with a snippet. Also check whether the site is included in these features in Search Console, following Google’s current instructions. Meeting the requirements does not guarantee that the page will appear.
Step 4. Account for each bot’s role
The label “AI bot” is too broad for access rules. Search, model training, and retrieval at a user’s request use different mechanisms.
| Bot or token | Purpose | How to interpret a restriction |
|---|---|---|
| Googlebot | Crawling for Google Search | Applies to search crawler access, including the basis for Search AI features |
| Google-Extended | Use of content for Gemini training and grounding in specified Gemini/Vertex AI products | A standalone control token, not a separate HTTP User-Agent; it does not affect inclusion or ranking in Google Search |
| OAI-SearchBot | ChatGPT search | Controls automated search crawling separately from training |
| GPTBot | Collecting content for possible OpenAI model training | Blocking training does not mean OAI-SearchBot is blocked |
| ChatGPT-User | Retrieval in response to user actions | Not automated search crawling; robots.txt may not apply |
| PerplexityBot | Search and links in Perplexity results | The operator separates it from collection for foundation model training |
| Perplexity-User | Retrieval at a user’s request | A separate mechanism that, according to the operator’s documentation, usually ignores robots.txt |
Check bot roles and current verification methods in Google’s bot list, OpenAI’s documentation and Perplexity’s help center. Don’t apply one provider’s policy to another.
For example, if the site owner has decided to allow ChatGPT’s automated search and block GPTBot’s collection, the relevant groups could look like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /This illustrates the separation of roles; it is not a ready-to-use robots.txt for your site. Before applying it, reconcile the groups with restricted paths and other rules. Allowing search does not require opening private pages. Blocking OAI-SearchBot also does not rule out all possible navigation links to the domain.
Step 5. Confirm actual access through the CDN and server
Check whether the request can get through the WAF, anti-bot protection, and rate limits. In server logs, match the time, path, status code, User-Agent, and IP. Make sure the response contains the document, not just a successful response from a protection check.
Verify that the bot belongs to the operator using the operator’s procedure or the published addresses in its documentation. A matching User-Agent string alone is not enough to justify an allow rule: it can be spoofed. Perplexity’s WAF instructions, for example, combine the User-Agent with current IP ranges.
If there are no actual records, write “actual request not confirmed.” No record may mean that the bot has not visited yet, not that access is blocked. After changing the CDN, repeat the external test and wait for an observable request; don’t declare the issue resolved based on a single local curl.
Step 6. Compare the source HTML and rendered page
Find the main fact in the original HTTP response, then in the DOM after JavaScript, and on screen. If the source contains only a shell, check how the relevant system retrieves the content. Don’t assume that every search or user-initiated retrieval system executes JavaScript like your browser.
Google can process JavaScript. For Google, check the result in the Search Console URL Inspection tool, and check resource and API availability. An initial noindex can cause rendering to be skipped; removing this directive with a script is not a reliable way to make a page accessible. This is described in Google’s JavaScript SEO documentation.
For other operators, confirm the specific retrieval method. SSR or static delivery of the main text may make retrieval easier, but there is no universal rule that “all AI sees only the source.” Links should lead to accessible pages containing the relevant facts; a pop-up after login is not a substitute for a public source.
For a detailed discussion of dynamic pages, see the guide to JavaScript and indexing. Text accessible at a URL does not by itself prove indexing: check the status separately in webmaster tools.
Step 7. Make the content verifiable
For each fact you selected, state the conditions, date, and source. A price without a currency, time period, and offer details may be unsuitable for comparison. A company name without a domain and region can easily be confused with a similarly named organization. Resolve contradictions between the service page, pricing, contact details, and the company’s own profiles.
Write the answer in the context of the reader’s task. A service page needs scope, limitations, and a next step; a guide needs reproducible steps, actual observations, and primary sources. Identify the real author and who is responsible for the material, keeping the publication date separate from the date of a substantial update. Don’t invent clients, biographies, or growth percentages.
Structured data must match the visible content. A separate “AI schema” is not required for Google Search, and llms.txt does not affect its visibility: Google explicitly says it ignores this file. See Google’s current guidance. For other systems, treat llms.txt as an experiment with a confirmed consumer, not a universal SEO task.
Keep FAQs when they answer real questions. Google stopped showing FAQ rich results on 7 May 2026 — official change log. FAQPage markup is not grounds for promising a special snippet or CTR increase.
Step 8. Measure results separately from readiness
Save the exact query, service, available mode, date, full answer, and links. Check whether a name refers to the right company; separately record a mention, a citation of your own URL, a link to a third-party source, and a visit in analytics. A provider error and a check that has not yet been completed are not the same as an answer without a mention.
After changing a page, repeat the same series of queries. A new answer is an observation: it may have changed because of updated sources, mode, or model. Drawing a conclusion about the effect of an edit requires comparable data; a single coincidence in timing does not prove causation.
Search Console provides the Generative AI performance report (Search) for impressions in Google AI features. The official help page reports a global launch on 31 August 2026; the report may be unavailable if there are too few impressions. It lets you analyze data by page, date, country, and device. Whether the report is available in the reChecker account has not been checked here. See Google’s help page.
This report does not measure all answers from ChatGPT or other providers. Check visits and completed actions in your site analytics; a mention alone is not a lead.
What reChecker can help you check
The reChecker technical audit performs 29 typical checks on selected URLs. It helps find technical obstacles, including page responses, directives, canonical, content, and links within the method’s scope. For a specific task, you can use the robots.txt analyzer, meta tag checker, and canonical checker.
To repeat the check after making a change, save the exact URL, time, and available report, then check the same address using the same method. If the redirect has changed, record the new final URL separately: different documents cannot be compared as if they were one page.
| Check | What to save | What conclusion it supports |
|---|---|---|
| Server responses and redirects | Status code, redirect chain, and final URL | Whether a response was received at the expected address; content is checked separately |
| robots.txt | The file, the crawler group, and the relevant path | Whether a crawl restriction applies |
| Meta tags and canonical | The directives received and the specified canonical version | Whether there is a technical inconsistency in the HTML checked |
| Technical audit | Check statuses and issues found | Which obstacles are confirmed by this method and what remains unknown |
| Search Console and server logs | URL status, crawl date, rendered page, and confirmed requests | How Google processed the page and whether real crawlers visited it |
A successful HTTP response does not confirm permission to index, and the absence of restrictions does not mean a URL is in the index. For Google AI Overviews and AI Mode, the page must also be indexed and eligible to appear with a snippet; this status is determined using Google data, not a reChecker technical score. The diagnostic process is described in Diagnosing a page that is not indexed.
reChecker does not have a separate crawler matrix or a special AI-readiness score. These checks do not confirm an actual visit by each provider, the URL’s presence in a search index, or the likelihood of a future citation. Server logs, rendering, and webmaster tool data supplement the automated report.
With a subscription, you can monitor up to five questions per month across seven services: ChatGPT, Alice, DeepSeek, Gemini, Claude, Grok, and Perplexity. These are results from a selected set of queries, not coverage of all user questions and not a separate measurement of Google AI Overviews. See pricing for payment terms.
Final check summary
URL и назначение:
Факт, который должен быть доступен:
Дата / время проверки:
Конечный HTTP-ответ и полученное содержание:
Директивы robots / noindex / snippet:
Исходный HTML / DOM / видимый экран:
Робот, его роль и применимые правила:
Реальный запрос подтверждён в логах: да / нет / не проверено
Источник и дата опубликованного факта:
Вопрос / сервис / ответ / ссылки:
Подтверждённое препятствие:
Владелец и действие:
Критерий повторной проверки:First, fix any reproducible obstacle: an incorrect response, an unintended directive, empty content, or an outdated fact. Repeat the same technical test, then assess the saved responses separately. To start with the current state of your pages, run a technical audit. To track recurring questions and changes, compare monitoring.