AI crawler access is easy to discuss and surprisingly difficult to verify. A robots.txt file may appear permissive while a content delivery network blocks the request. A crawler may receive a 200 status but only see an empty application shell. A vendor may document one user agent while your logs show several related request patterns. None of those findings, on its own, proves that a page will appear in an AI-generated answer.
That is why an AI crawler access audit for SEO teams should be treated as an access-control and observability exercise, not as a shortcut to AI visibility. Its job is to establish which systems can request which resources, what they actually receive and whether that outcome matches the organisation’s policy.
The distinction matters. Access can support discovery, indexing, retrieval or citation, but it does not guarantee any of them. Content quality, relevance, authority, product design and platform-specific selection systems remain separate questions.
This guide lays out the audit process I use as a practical baseline. It prioritises first-party documentation, reproducible tests and server-side evidence over crawler lists copied from third-party blog posts.
What an AI crawler access audit should answer
A useful audit answers five questions:
- Which automated agents matter to the business? This includes conventional search crawlers, documented AI crawlers and agents that retrieve a page in response to a user action.
- What access policy does the organisation intend? Teams may want search indexing, AI answer visibility, user-triggered retrieval, model-training access or different rules for each.
- What do published controls say? Review robots.txt, page-level robots directives, response headers, authentication rules and relevant platform settings.
- What happens at the edge and origin? CDN rules, bot management, rate limits, geofencing and application security can override the practical effect of a permissive robots.txt file.
- What evidence shows that requests occurred? Logs can reveal attempted access, response codes and resource paths, although they cannot prove downstream use or citation.
The output should not be a binary “AI-ready” score. It should be an agent-by-agent decision register with evidence, uncertainty and an owner for each corrective action.
Start with policy, not a list of bot names
Many audits begin by collecting every user-agent string mentioned online. That creates activity without resolving the core business decision.
First decide what types of use the organisation is prepared to permit. Legal, security, publishing and commercial teams may have different concerns. SEO should coordinate the decision, but it should not silently make policy for the entire organisation.
| Access purpose | Question to resolve | Typical evidence |
|---|---|---|
| Search discovery and indexing | Should eligible public pages remain available to established search crawlers? | Search documentation, robots.txt, index status and server logs |
| AI search or answer retrieval | Should documented systems be able to retrieve pages for answers or citations? | First-party bot documentation, live fetch tests and logs |
| User-triggered access | Should an agent be able to fetch a URL when a user explicitly requests it? | Vendor documentation, request patterns and security policy |
| Model development or training | Does the organisation permit this use, and are technical controls available? | Current first-party terms and documentation, reviewed by appropriate specialists |
| Restricted content | Which resources must remain unavailable regardless of crawler identity? | Authentication, authorisation and data-classification controls |
Do not use robots.txt to protect confidential material. It is a public set of crawl instructions, not an access-control system. Sensitive resources need authentication and server-side authorisation.
Use an explicit evidence hierarchy
AI crawler information changes. User agents are renamed, products are combined and controls can acquire narrower meanings. A durable audit therefore records the source and retrieval date for every platform-specific claim.
I use the following hierarchy:
- Current first-party documentation for the named product or agent.
- Observed server, CDN and security logs from the audited property.
- Controlled request tests that reproduce the response delivered by the site.
- Vendor support confirmation retained in writing when public documentation is ambiguous.
- Third-party research as a lead to investigate, not as final proof.
Google’s documentation says that the technical requirements for appearing in its AI search features are the existing requirements for Search, including indexability and snippet eligibility; it does not prescribe a separate AI-specific optimisation file. Review the current Google Search documentation for AI features rather than relying on screenshots or summaries that may be outdated.
Perplexity publishes a first-party reference describing its bots and recommended handling. Use the current Perplexity bot documentation when evaluating those agents. The important methodological point is broader: do not assume that discovery crawlers and user-triggered fetchers serve the same purpose or obey identical controls. Verify each documented agent separately.
Avoid assigning meaning to an unfamiliar user agent based only on its name. User-agent strings can be spoofed. Where a vendor supplies an IP-verification method, test it exactly as documented and retain the result. Where no reliable verification exists, label attribution as uncertain.
Build an access inventory before testing
Create a compact inventory of the property’s public and restricted surfaces. Large sites rarely apply one consistent rule everywhere.
Include:
- Primary HTML pages and important subdomains
- International or regional hosts
- JavaScript, CSS and image assets needed to understand a page
- PDFs, feeds and other indexable documents
- API endpoints used to render public content
- Staging, account, checkout and internal search areas
- Robots.txt and XML sitemap locations
- CDN, WAF and bot-management products in the request path
Sample URLs by template and risk, not only by traffic. A homepage test says little about a documentation subdomain, a faceted category page or an article rendered through a separate application stack.
For each sample, record the intended audience, canonical status, indexability, authentication requirement and desired agent policy. This prevents the team from “fixing” a block that was deliberate.
Run the audit in seven practical steps
1. Capture all public crawl directives
Download robots.txt directly from every relevant host. Store the raw file with the test date and response headers. Check status codes, redirects, caching and whether different regions receive different versions.
Then inspect page-level meta robots directives and X-Robots-Tag headers. These controls usually address indexing or presentation rather than network access, so do not treat them as interchangeable with robots.txt.
Look for:
- Broad disallow rules that unintentionally cover editorial content
- Agent-specific groups that conflict with general groups
- Rules copied from production to staging or vice versa
- Blocked rendering resources
- Old directives retained after a CMS or domain migration
- Headers added by the CDN rather than the application
Evaluate rule matching with a parser that follows the documented behaviour of the crawler being assessed. A generic robots tester can be useful, but it is not authoritative for every agent.
2. Test the real response path
A text review of robots.txt is not enough. Request representative URLs through the public hostname and inspect the final response.
curl -I https://www.example.com/important-page/
curl -L -s -D headers.txt -o page.html https://www.example.com/important-page/Repeat controlled tests with the user-agent strings under review, but interpret the results carefully. Changing a command-line user agent does not prove identity and may not reproduce a verified crawler’s network location. It can still reveal obvious user-agent-based blocking.
Record the final URL, status code, response size, content type, robots headers, cache result and any challenge page. Save the returned body. A 200 response containing “Access denied” is not successful content access.
3. Inspect CDN, WAF and bot-management rules
This is where many nominally open sites become inaccessible. Security systems may challenge non-browser traffic, block cloud hosting ranges, enforce request-rate thresholds or apply managed bot categories that nobody on the SEO team has reviewed.
Ask the infrastructure owner for the active rules, not only screenshots from a default dashboard. Check:
- Automated bot classifications
- Custom user-agent blocks
- JavaScript or cookie challenges
- IP reputation and hosting-provider rules
- Geographic restrictions
- Rate limits and temporary bans
- Origin rules that differ from CDN behaviour
Do not immediately allowlist an agent globally. Prefer the narrowest rule that achieves the intended access, and retain protections for login, account and transaction paths.
4. Analyse logs without overclaiming
Server and edge logs provide stronger evidence than a robots.txt inspection because they show requests that reached your systems. They still require careful interpretation.
For each candidate agent, examine:
- First and most recent observed request
- Requested hosts, directories and file types
- Status-code distribution
- Redirect chains
- Response bytes and latency
- Robots.txt request behaviour
- Repeated failures, challenges or rate-limit responses
- IP verification status, if a documented method exists
A request with a recognisable user agent proves only that a request bearing that string reached the system. It does not prove that the vendor sent it. Nor does a successful fetch prove indexing, model use, citation or answer inclusion.
Absence from logs is also ambiguous. The system may not have attempted to crawl the site during the retention window, logs may be incomplete, or requests may have been blocked upstream. State those limitations in the audit.
5. Verify that accessible pages contain usable content
Network access is only the first layer. Inspect the HTML returned before client-side execution. Confirm that the page exposes its primary heading, answer-bearing copy, links and relevant entity information in a stable form.
Modern search systems can process JavaScript to varying degrees, but an audit should not assume every retrieval system runs the same rendering pipeline. If the initial HTML contains little more than an application shell, record that as a dependency and test the rendered output separately.
Also review structured data for accuracy. It can help eligible search systems understand page entities and attributes, but it is not a substitute for visible content and does not guarantee enhanced treatment. Google’s structured data introduction is the appropriate first-party starting point for its Search implementation.
For broader editorial issues, pair the technical review with an AEO content audit. Crawler access cannot rescue a page that does not answer the relevant question clearly or support its claims.
6. Separate access findings from visibility findings
This separation prevents misleading reporting. I recommend three independent statuses:
- Access status: allowed, blocked, challenged, conditional or unknown
- Content status: complete, partial, rendered-only, restricted or unsuitable
- Visibility status: observed, not observed or not measured
Visibility needs its own method, query set and observation period. The guidance in How to Measure Visibility in AI Search Without Misleading Metrics explains why citations, mentions and answer presence should not be collapsed into a single inflated score.
Likewise, being technically available does not mean a source is citation-worthy. If citation is the goal, assess the page’s evidence, specificity, freshness and attribution using a workflow such as How to Get Cited by ChatGPT, Perplexity and Google AI.
7. Produce an agent-by-agent decision register
The final register should be understandable to SEO, security, legal and engineering teams. Avoid a vague recommendation to “unblock AI bots.”
| Field | What to record |
|---|---|
| Agent or system | Exact documented name and user-agent string, where available |
| Claimed purpose | Discovery, indexing, answer retrieval, user-triggered fetch or another documented function |
| Source | First-party URL and date checked |
| Intended policy | Allow, restrict, block or pending review |
| Observed result | Status, response body, challenge and sampled paths |
| Confidence | Verified, probable or uncertain, with the reason |
| Action | Specific configuration or investigation required |
| Owner | SEO, platform, security, legal or content |
Prioritise contradictions. If policy says “allow” but the WAF challenges every request, that is actionable. If policy itself is undecided, configuration work should wait until the appropriate owner resolves it.
Common audit mistakes
Treating every AI product as one crawler
Products may use different systems for search discovery, model development and user-requested retrieval. Never infer that one directive governs all product functions unless current first-party documentation says so.
Assuming robots.txt is an enforcement boundary
Compliant crawlers may observe published directives, but robots.txt does not prevent arbitrary clients from requesting public URLs. Use authentication for protected information.
Reading success from status codes alone
Challenge pages, consent interstitials and soft blocks can return 200. Inspect response bodies and compare them with a normal browser response.
Blocking rendering assets indiscriminately
If a page depends on blocked scripts or APIs to expose its main content, some systems may receive an incomplete representation. Test the dependency before changing rules; not every asset needs to be crawlable.
Promising visibility after access is fixed
Removing an accidental block restores an opportunity to fetch. It does not guarantee crawling frequency, inclusion, citation, rankings, leads or revenue. Report the fix in those limited terms.
How to monitor access after the audit
Turn the one-off review into a lightweight control. Recheck robots.txt and important response headers after CDN, CMS, security or domain changes. Alert on unexpected status shifts for a small set of representative public URLs.
Maintain a version-controlled policy file or decision register. Each entry should include the evidence source and review date. Revalidate platform-specific claims against current first-party documentation rather than automatically carrying last quarter’s assumptions forward.
Log retention should be long enough to investigate infrequent requests, subject to the organisation’s privacy and security policies. Dashboards can track verified or candidate requests, but they should preserve an “unknown” category rather than forcing uncertain traffic into a named vendor bucket.
FAQ
Does allowing an AI crawler improve AI search visibility?
It may remove a technical barrier to discovery or retrieval, depending on the system. It does not guarantee inclusion, citation or prominent visibility.
Should SEO teams allow every documented AI bot?
No. Access should follow an explicit business policy informed by current documentation, security requirements and the purpose of each agent.
Is robots.txt enough for an AI crawler access audit?
No. Review page directives, CDN and WAF rules, server responses, rendering dependencies and logs. Robots.txt describes intended crawl behaviour but does not reveal every practical block.
Can server logs prove that an AI platform used our content?
Logs can show requests and responses. They generally cannot prove downstream indexing, training, answer generation or citation. User-agent attribution may also require verification.
How often should the audit be repeated?
Repeat key tests after infrastructure or policy changes and review the full register periodically. The appropriate interval depends on release frequency, risk and the stability of the site’s delivery stack.
Conclusion: audit access as a control, not a visibility promise
A credible AI crawler access audit gives SEO teams a defensible account of intent, configuration and observed behaviour. It identifies where public directives, edge controls and actual responses disagree. Just as importantly, it documents what the evidence cannot establish.
Start with a policy matrix. Verify agent claims against current first-party sources. Test representative URLs through the real delivery path, inspect response bodies and use logs with appropriate attribution caveats. Then assign narrow, owned actions for genuine contradictions.
The immediate objective is specific: ensure that systems the organisation intends to admit can reach the intended public content, while restricted resources remain protected by real access controls. Measure AI search visibility separately. That boundary keeps the audit useful, technically honest and repeatable.
