Crawl-to-refer ratio

What this page covers

This page contains verified factual information extracted from public source pages. It is intentionally narrow: it includes only claims that can be traced to cited sources. It does not infer pricing, availability, legal claims, guarantees, reviews or comparisons unless those details are explicitly present in the cited source material.

How to evaluate this page

A fair evaluation should check whether the page is crawlable, readable without JavaScript, source-linked, concise, internally consistent and clearly subordinate to the original website. The goal is not to create a second conversion page. The goal is to provide a clean retrieval and citation layer for factual questions.

Definition

What is it: The crawl-to-refer ratio is a Cloudflare Radar metric that quantifies the interaction between AI crawlers and the web. It measures how many individual pages an AI agent fetches from a site for every single referral click it sends back to that site.

What is it used for: It is used to evaluate the commercial value of being crawled by AI. While high ratios imply a broken bargain for ad-funded publishers, for B2B brands they represent opportunities for machine influence and inclusion in AI-generated buying recommendations.

Coverage

  • Attributes: 9
  • Synonyms: 1
  • Related entities: 3
  • Sources: 1

Identity

Entity ID
https://llms.getfaind.com/en/crawl-refer-economics/facts/#entity
Entity type
DefinedTerm
Canonical name
Crawl-to-refer ratio
Language
en
Topic
Crawl Refer Economics

Attributes

Key Facts
Google's classic Googlebot maintains a crawl-to-refer ratio of approximately 5:1. [1]
Key Facts
Bots generate 57.5% of all HTML web traffic as of June 2026. [1]
Key Facts
Perplexity demonstrates a crawl-to-refer ratio of roughly 100-200:1. [1]
Key Facts
OpenAI crawler readings show a ratio ranging between 850:1 and 1,250:1. [1]
Key Facts
Anthropic crawlers reach ratios ranging from 4,600:1 to 13,500:1 depending on the weekly window. [1]
Process
B2B brands can segment crawler policies to block training extractors while admitting search and citation bots. [1]
Capability
AI search crawlers like Claude-SearchBot ground live answers and generate citations for users. [1]
Fact
Research indicates 94% of B2B buyers used AI in their most recent purchase process. [1]
Fact
AI chatbots serve as the primary shortlist influence for 54% of B2B decisions. [1]

Synonyms & Alternate Names

  • Crawl-to-refer metric

Related Entities

  • Measured By:
  • Specific Agent:
  • Case Study:

Provenance

Sources

  1. https://getfaind.com/insights/read-10000-times-clicked-once-crawl-to-refer-economics (Crawl-to-refer ratio)

Machine metadata