A few years ago, most small business websites had one big crawler to care about: Googlebot.
Now your server logs can include GPTBot, ClaudeBot, Claude-SearchBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, Perplexity-User, Applebot-Extended, Google-Extended, Meta-ExternalAgent, Bytespider, CCBot, and plenty of crawlers that don’t announce themselves cleanly.
That changes the job. Your website isn’t just being read by prospects, customers, and search engines anymore. It’s also being scraped for model training, fetched for AI answers, checked by user-triggered AI tools, and hit by automated systems that may never send a buyer back.
This resource collects the AI crawler statistics that matter most in 2026. Use it to explain the issue to a client, update a website governance plan, or decide whether your site should allow, limit, monitor, or block certain AI crawlers.
Quick AI crawler statistics for 2026
Automated traffic is now the majority of the web. Imperva’s 2026 Bad Bot Report says automated traffic accounted for more than 53% of all web traffic in 2025, up from 51% in 2024, while human activity fell to 47%. (Imperva)
AI crawling is a major part of that machine traffic shift. Cloudflare reported that search and AI crawler traffic was 32% higher year over year in April 2025, 24% higher in June, and still 4% higher in July after the spring spike cooled. (Cloudflare)
Training is the biggest AI crawler use case. Cloudflare’s AI Insights work found that training traffic was responsible for nearly 80% of crawling from AI bots, while user-action and undeclared purposes each represented much smaller shares during the measured period. (Cloudflare)
GPTBot’s share of AI crawling more than doubled in Cloudflare’s year-over-year data, moving from 4.7% to 11.7%, while ClaudeBot rose from 6% to nearly 10%. (Cloudflare)
Some crawlers fell just as fast. Cloudflare’s data showed ByteDance’s Bytespider dropping from 14.1% to 2.4% of AI-only crawler traffic, and Amazonbot falling from 10.2% to 5.9%. (Cloudflare)
AI tools are sending more visits than they used to, but still not much compared with Google. SE Ranking found AI search engines generated 0.32% of total website traffic in 2026, up from 0.24% in 2025 and 0.02% in 2024. (SE Ranking)
That means AI referral traffic grew 16x from 2024 to 2026, but organic search still accounted for 42.75% of all website traffic in SE Ranking’s dataset. (SE Ranking)
The crawl-to-click problem is the real story
For business owners, the issue isn’t that a bot visited the site. Google has crawled websites for decades, and most businesses happily accepted that trade because Google sent searchers back.
AI crawlers changed the math. Some bots consume a lot of pages and send back very little referral traffic.
Cloudflare tracks this as a crawl-to-refer ratio, which compares HTML page requests from a crawler with HTML page requests referred by that platform. In July 2025, Cloudflare found Anthropic still had a ratio of 38,000 crawls per visitor, even after improving from 286,000:1 in January. (Cloudflare)
Perplexity’s July 2025 ratio was 194 crawls per visitor, and Cloudflare said Perplexity had more crawling but fewer referrals during the measured period. (Cloudflare)
The ratio can vary by industry. In a later Cloudflare breakdown, Anthropic’s crawl-to-refer ratio for one vertical was 8,800:1, OpenAI’s was 401.7:1, and Perplexity’s was 88:1, which shows why site owners should compare their own logs against their own business model instead of assuming one web-wide average applies. (Cloudflare)
That matters because crawling has a cost. A small brochure site may not feel it. A site with thousands of product pages, faceted URLs, internal search results, PDFs, image assets, and old blog posts can spend real hosting, CDN, and engineering time serving requests that don’t turn into leads.
AI Overviews are changing the value of being cited
Getting cited by an AI answer can still be valuable. A good citation can create brand trust, especially for expert content, local services, technical explainers, and buying guides. The problem is that a citation doesn’t always produce a click.
Pew Research Center analyzed 68,879 unique Google searches from March 2025 browsing data and found that 18% generated an AI summary. (Pew Research Center)
When users saw an AI summary, they clicked a traditional search result in 8% of visits. When they did not see an AI summary, they clicked a traditional result in 15% of visits. (Pew Research Center)
Clicks inside the AI summaries were even rarer. Pew found users clicked a link in the AI summary in just 1% of visits to pages with a summary. (Pew Research Center)
Users were also more likely to stop browsing after seeing an AI summary. Pew found browsing sessions ended after 26% of pages with an AI summary, compared with 16% of pages with only traditional search results. (Pew Research Center)
Longer and question-based searches are more exposed. Pew found only 8% of one- or two-word searches produced an AI summary, but 53% of searches with 10 words or more did. Searches beginning with question words produced AI summaries 60% of the time. (Pew Research Center)
Semrush’s 2025 AI Overviews study gives another angle. Across 10 million-plus keywords, AI Overviews appeared for 6.49% of tracked queries in January 2025, rose to 24.61% in July, then settled at 15.69% in November. (Semrush)
The same Semrush study found the intent mix changed during 2025. Informational queries made up 91.3% of queries that triggered AI Overviews in January, but only 57.1% by October, while commercial, transactional, and navigational AI Overviews grew. (Semrush)
For a business website, that means AI search is no longer just a top-of-funnel publishing issue. It can touch product comparisons, service pricing, branded searches, and local buying questions.
AI referrals are small, but more qualified when they arrive
AI search traffic is still small in analytics, but the visitors who do arrive may be worth watching.
SE Ranking found AI platforms accounted for 0.32% of total website traffic in 2026, which is roughly 1 in every 312 website visits. (SE Ranking)
The same research found visitors referred by AI engines spent 68% more time on websites than visitors from traditional organic search. (SE Ranking)
ChatGPT is the largest AI referral source in most datasets, but not every study gives it the same share. SE Ranking found ChatGPT had 74.78% of AI referral traffic in 2026, followed by Gemini at 11.56%, Perplexity at 7.23%, Copilot at 3.51%, and Claude at 2.62%. (SE Ranking)
Previsible’s dataset, reported by Search Engine Land, analyzed 6.77 million LLM-driven sessions and found ChatGPT commanded 92.4% of trackable LLM referral traffic, with monthly LLM sessions growing from 65,249 in November 2024 to 644,478 in May 2026. (Search Engine Land)
Those differences are a useful warning. AI referral reporting depends on the dataset, region, vertical, bot labeling, attribution rules, and whether Google AI Overviews are counted separately from standalone AI tools. Your own numbers matter more than the headline average.
Robots.txt adoption shows site owners are reacting
Robots.txt doesn’t physically block a crawler. It states instructions. Respectful crawlers follow them, and bad actors ignore them.
Still, robots.txt data shows how quickly website owners changed their posture toward AI scraping.
Paul Calvano’s HTTP Archive analysis found 94.12% of 12,155,217 websites in July 2025 served a robots.txt file containing at least one directive. (Paul Calvano / HTTP Archive)
GPTBot rules appeared quickly after OpenAI published crawler documentation. Calvano found the number of sites including rules for GPTBot went from 0 to almost 125,000 in August 2023, reached 299,000 a month later, and hit 578,000 by November 2023. (Paul Calvano / HTTP Archive)
By 2025, ChatGPT, Claude, Facebook, and other AI-related user agents appeared in the robots.txt files of more than 560,000 sites. (Paul Calvano / HTTP Archive)
Popular websites are more likely to manage AI bot access. Calvano found nearly 21% of the top 1,000 websites had rules for GPTBot in robots.txt. (Paul Calvano / HTTP Archive)
The catch is maintenance. New AI user agents keep appearing, and different platforms use separate crawlers for training, search, and user actions. OpenAI lists ChatGPT-User, GPTBot, and OAI-SearchBot; Anthropic lists ClaudeBot, Claude-User, and Claude-SearchBot; Apple lists Applebot-Extended. (OpenAI, Anthropic, Apple)
That means a one-time robots.txt edit can go stale. Treat AI crawler rules like website security rules. Review them, test them, and update them when your strategy changes.
What website owners should do with this data
The smart answer is not “block every AI bot” or “allow every AI bot.” The right answer depends on your business.
A publisher that makes money from pageviews has a different risk profile than a contractor who wants AI tools to recommend their service pages. An ecommerce store may want product data visible to buying assistants but may not want internal search pages, staging URLs, or filtered category pages crawled heavily. A B2B firm may want explainers cited by ChatGPT and Claude but may still want to limit training crawlers that don’t send meaningful traffic.
Start with your logs. Cloudflare, server logs, CDN logs, and analytics can show which AI user agents are requesting pages, how often they hit your site, which pages they request, and whether those platforms send back visitors.
Then separate crawler purposes. Training crawlers, search crawlers, and user-action crawlers are not the same thing. Cloudflare’s crawl-purpose breakdown separates Training, Search, User action, and Undeclared, which is a practical model for business decisions. (Cloudflare)
Next, protect wasteful surfaces. Internal search results, faceted ecommerce pages, duplicate parameter URLs, thin tag archives, old staging paths, and oversized media libraries can turn into crawler traps. If those pages don’t help buyers or AI answers, don’t let bots spend your crawl budget there.
Finally, improve the pages that AI systems should understand. Clear service pages, pricing ranges, FAQs, product specs, author bios, review signals, location details, and schema markup help both humans and machines understand what you do. That doesn’t guarantee citation, but it gives AI systems less room to guess.
A practical AI crawler policy for a small business site
Use this simple framework when you’re deciding what to allow.
Allow discovery where it can create demand
Keep important service pages, product pages, educational guides, location pages, case studies, and high-trust pages accessible to mainstream search crawlers. If AI answer engines cite your business, these are the pages most likely to create qualified demand.
Limit crawling where it creates cost without value
Disallow or noindex low-value surfaces such as search result pages, filtered URLs, duplicate archives, cart pages, account pages, test environments, and large generated URL sets. Google has long warned that internal search result pages can create search quality problems, and the same logic applies when AI crawlers hit them at scale. (Google Search Central)
Review AI-specific user agents quarterly
Crawler names change. Policies change. Referral behavior changes. Cloudflare’s data shows crawler shares can move sharply within a year, including GPTBot, ClaudeBot, Bytespider, and Amazonbot. (Cloudflare)
A quarterly review is enough for most small businesses. High-traffic publishers, SaaS companies, and ecommerce sites should review monthly.
The bottom line
AI crawlers are not a side issue anymore. Bots made up more than 53% of web traffic in 2025, AI training made up nearly 80% of AI bot crawling in Cloudflare’s data, and AI referral traffic is growing even though it remains a small share of total visits. (Imperva, Cloudflare, SE Ranking)
The websites that handle this well won’t be the ones that panic. They’ll be the ones that measure crawler activity, protect low-value surfaces, make high-value pages easier to understand, and keep their rules current.
If you want help auditing how AI crawlers, search engines, and real customers interact with your site, start a project with Your Web Team. We’ll help you turn the data into a website plan that protects your content and still brings in leads.
