Most website owners think about one crawler: Googlebot. They configure their robots.txt for it, check their indexing status in Google Search Console, and measure success by Google rankings.
Meanwhile, their site is being visited — regularly, silently — by GPTBot, PerplexityBot, ClaudeBot, OAI-SearchBot, and Bingbot. Each of these crawlers powers a different AI search product. Each has different capabilities. Each makes different decisions about what content to use and what to ignore.
Optimizing for Googlebot alone is no longer sufficient. This article maps how each major crawler works, where they differ, and what that means for building a site that is visible across all of them.
How Googlebot Processes Your Website
Googlebot is the most sophisticated web crawler in existence. Understanding its full workflow reveals why it remains the benchmark — and why even its sophistication creates specific problems that AI crawlers do not share.
Step 1: Discovery
Googlebot discovers URLs through three main channels:
- XML sitemaps submitted via Google Search Console
- Internal links followed from already-known pages
- External links from other sites pointing to yours
A URL that does not appear in a sitemap and has no links pointing to it is effectively invisible to Googlebot. Discovery is the prerequisite for everything that follows.
Step 2: Fetching
When Googlebot visits a URL, it sends an HTTP GET request. Before fetching, it checks your robots.txt file to confirm the URL is allowed. If disallowed, it stops immediately — no fetch, no index.
The fetch retrieves whatever the server returns: typically an HTML document, plus the headers. This raw server response is what Googlebot receives first.
Step 3: Rendering — The Two-Wave System
This is where Googlebot diverges most significantly from AI crawlers, and where much of the confusion about JavaScript and SEO originates.
Wave 1 — Immediate: Googlebot processes the raw HTML from the server response right away. Any content present in that initial HTML is indexed immediately. This is fast, reliable, and consistent.
Wave 2 — Delayed: Googlebot queues the page for full JavaScript rendering. A headless Chromium instance eventually executes the page’s JavaScript, builds the complete DOM, and that rendered version is indexed. This wave can happen hours, days, or even weeks after Wave 1 — and lower-priority pages may wait longer.
The consequence: content that only exists after JavaScript execution may be indexed eventually, but not quickly. For time-sensitive content, or for sites that rely entirely on JavaScript rendering, this delay is a genuine competitive disadvantage.
Step 4: Indexing
After rendering, Googlebot’s systems analyze the content — text, headings, links, images, structured data, metadata — and store it in Google’s search index. The page is associated with topics, entities, and queries it appears relevant to.
Not every fetched page is indexed. Pages Google judges as low-quality, thin, duplicated, or irrelevant may be crawled but not stored. The noindex directive can also explicitly prevent indexing.
Step 5: Ranking
When a user queries Google, the ranking system evaluates all indexed pages relevant to that query and assigns positions based on hundreds of signals — E-E-A-T, PageRank, Core Web Vitals, relevance, freshness, and many more.
Googlebot’s defining characteristic: it is patient, sophisticated, and persistent. It will retry pages, attempt JavaScript rendering, re-evaluate content over time. It is the most forgiving crawler — and even it has limits that simpler AI crawlers do not approach.
How AI Crawlers Process Your Website
AI crawlers share a common workflow that is simpler than Googlebot’s — and that simplicity is the key fact that shapes everything about AI visibility optimization.
The Common AI Crawler Workflow
- Receive a URL to fetch (from a sitemap, a link, or a retrieval query)
- Send an HTTP GET request
- Check
robots.txtfor permission - Receive the server’s HTML response
- Extract text content from the HTML
- Store or use that content (for training or for real-time answer generation)
That is the complete workflow. There is no rendering queue. There is no headless browser. There is no Wave 2. What arrives in the HTTP response is what the crawler sees — nothing more.
GPTBot (OpenAI)
GPTBot serves two purposes: collecting training data for OpenAI’s language models, and supporting real-time retrieval for ChatGPT Search through its companion crawler OAI-SearchBot.
- Does not execute JavaScript
- Respects
robots.txt— you can block it withUser-agent: GPTBot / Disallow: / - Identifies itself via user-agent string
GPTBot - Crawl frequency: periodic, not continuous
Blocking GPTBot prevents your content from being used in OpenAI model training but does not block OAI-SearchBot (ChatGPT Search retrieval). These are separate bots with separate user-agent strings — a distinction most site owners don’t realize exists.
OAI-SearchBot (OpenAI)
OAI-SearchBot is OpenAI’s real-time retrieval crawler, specifically powering ChatGPT Search. When a user asks ChatGPT Search a question, OAI-SearchBot fetches relevant pages at query time to inform the answer.
- Does not execute JavaScript
- Operates in near-real time at query time
- Speed-sensitive — slow pages may time out and be skipped
- Separate from GPTBot: blocking one does not block the other
PerplexityBot (Perplexity AI)
PerplexityBot is a pure retrieval crawler — Perplexity AI does not train its own foundational model in the way OpenAI does. Every query triggers a real-time web fetch.
- Does not execute JavaScript
- Extremely speed-sensitive: Perplexity’s product promise is instant answers, so pages that respond slowly are deprioritized
- High crawl frequency relative to other AI bots — Perplexity queries happen continuously
- Fresh content receives priority:
<lastmod>dates in sitemaps influence which pages get recrawled
ClaudeBot (Anthropic)
ClaudeBot supports Anthropic’s model training and Claude’s web access capabilities.
- Does not execute JavaScript
- Respects
robots.txt - HTML-only, identical behavior pattern to GPTBot
Bingbot (Microsoft)
Bingbot powers both Bing’s traditional search index and Microsoft Copilot’s answer engine. Of all the AI-adjacent crawlers, Bingbot has the most Googlebot-like capabilities — it has limited JavaScript rendering ability, though far less sophisticated than Googlebot’s two-wave system.
- Limited JavaScript rendering (less capable than Googlebot)
- Powers both Bing rankings and Copilot citations
- Respects
robots.txt - Following standard Bing SEO practices largely aligns with Copilot visibility
Rendering Differences: The Critical Comparison
| Crawler | JS Execution | Rendering Model | Powered Product |
|---|---|---|---|
| Googlebot | Yes (delayed) | Two-wave | Google Search + AI Overviews |
| OAI-SearchBot | No | HTML-only | ChatGPT Search |
| GPTBot | No | HTML-only | OpenAI training |
| PerplexityBot | No | HTML-only | Perplexity AI |
| ClaudeBot | No | HTML-only | Anthropic training |
| Bingbot | Limited | Partial | Bing Search + Copilot |
The pattern is unambiguous: every AI-powered product except Google’s relies on HTML-only crawlers. JavaScript-rendered content is invisible to all of them.
This creates a stark implication. A site built with client-side rendering may:
- Eventually rank on Google (after Wave 2 rendering, which may take weeks)
- Never appear in ChatGPT Search answers (OAI-SearchBot sees nothing)
- Never appear in Perplexity answers (PerplexityBot sees nothing)
- Never appear in Claude’s web access responses (ClaudeBot sees nothing)
- Appear partially in Copilot (Bingbot’s limited rendering may catch some content)
Five products, five audiences, one architectural decision determining visibility across all of them.
Indexing Differences: Persistent vs Real-Time
Beyond rendering, the crawlers differ fundamentally in how they use what they collect.
Google’s Persistent Index
Googlebot feeds a persistent, continuously updated search index. When a page is crawled and indexed, it remains in the index until Googlebot recrawls and re-evaluates it. Rankings fluctuate as the index is refreshed, but indexed pages don’t disappear overnight.
This persistence means older content with strong authority signals can maintain rankings for years. Freshness matters for some queries but not all.
AI Training Crawlers
GPTBot and ClaudeBot collect content that eventually becomes part of model training datasets. This process is periodic and non-real-time — content gathered today contributes to future model training, not to answers being generated right now. There is a meaningful lag between a page being crawled and that content influencing model knowledge.
For most practical purposes, training crawlers are about the long-term knowledge base of the model, not immediate discoverability.
AI Retrieval Crawlers
OAI-SearchBot and PerplexityBot operate in a completely different mode: they fetch pages at the moment a user submits a query. There is no persistent index in the traditional sense — the retrieval system fetches, processes, and uses content on demand.
This real-time model has two implications:
Freshness is critical. A page updated today can be cited in a Perplexity answer today. A page with stale <lastmod> dates may be deprioritized in favor of more recently updated sources.
Speed determines inclusion. A page that takes 4 seconds to respond may time out during a real-time retrieval fetch. Crawl budget is effectively replaced by response-time tolerance — if your page doesn’t respond fast enough, it simply isn’t included.
Why the Same Page Can Rank on Google But Be Invisible to AI Systems
This is the scenario that confuses most site owners — and it has a precise explanation.
Consider a page built with client-side rendering:
- Googlebot Wave 1 fetches the empty HTML shell → nothing indexed yet
- Googlebot Wave 2 (days later) renders the JavaScript → full content indexed → page eventually ranks
- OAI-SearchBot fetches the empty HTML shell → no content extracted → page never cited in ChatGPT Search
- PerplexityBot fetches the empty HTML shell → no content extracted → page never cited in Perplexity
- ClaudeBot fetches the empty HTML shell → no content for training → page absent from Claude’s knowledge base
The page ranks on Google because Google invested the resources to render it. It is invisible to AI answer engines because they do not make that investment.
This is why Google rankings are no longer a reliable proxy for AI visibility. The two systems evaluate content through fundamentally different mechanisms, and a site architecture that satisfies Google’s two-wave process may completely fail the HTML-only requirement of every AI crawler.
Best Practices for Multi-Crawler Optimization
Optimizing for all crawlers simultaneously is simpler than it sounds, because all crawlers share a common requirement: complete, accessible HTML in the server response. Meeting the most demanding requirement (HTML-only AI crawlers) automatically satisfies all less demanding ones.
Deliver complete HTML in the server response. Use SSR or SSG for all public-facing content pages. No content should require JavaScript execution to appear. This single requirement addresses the crawlability needs of every crawler simultaneously.
Configure robots.txt deliberately. Review which crawlers you want to allow and which to block. By default, allow all. If you choose to block training crawlers (GPTBot, ClaudeBot), do so explicitly. Remember that blocking GPTBot does not block OAI-SearchBot — use separate directives for each.
Maintain an accurate XML sitemap. Include all important public pages. Keep <lastmod> dates accurate and updated when content changes — this is particularly important for PerplexityBot and other freshness-sensitive retrieval crawlers.
Implement JSON-LD schema in the server-rendered <head>. Structured data delivered server-side reaches every crawler. Schema injected by client-side JavaScript reaches none of the AI crawlers.
Optimize server response time. Target under 800ms TTFB before caching, under 200ms with CDN caching. Retrieval crawlers operating in real time have no tolerance for slow responses.
Monitor bot traffic in server logs. Check which crawlers are actually visiting your site. Look for GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, and Bingbot in your access logs. A crawler you have never seen in your logs is a product that has never indexed your content.
The Universal Requirement
Every crawler covered in this article — Googlebot, GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Bingbot — will successfully extract content from a well-structured, server-rendered HTML page.
Not a single one of them requires JavaScript execution to discover, extract, and use your content — as long as it is in the HTML.
Optimize for the simplest crawler. Every more capable crawler benefits automatically.
📈 Deep Dive: Anthropic AI Visibility Audit
Curious how these crawler mechanics apply to the world’s leading AI brands? This comprehensive technical blueprint analyzes anthropic.com and claude.ai across technical SEO, JavaScript rendering, and Generative Engine Optimization (GEO).
Access the Complete Anthropic.com SEO Audit →
Want to see exactly which crawlers are visiting your site — and what they’re finding? The Answer Engine Visibility Diagnostic checks your server logs pattern, robots.txt configuration, and HTML output against every major AI crawler. Delivered automatically, no discovery call required.
Next: The JavaScript SEO Myth: What Search Engines Can and Cannot Render →
← Previous: Why Server-Side Rendering Matters for Generative Engine Optimization
This article is part of a 20-article series on SEO, GEO, and AI Visibility. View the complete series →
Leave a Reply