Put a Price on Participating AI Crawler Access

Cloudflare Pay Per Crawl lets you charge participating AI crawlers per successful content retrieval without building metering, billing or payouts at your origin. You assign each listed crawler Allow, Charge or Block, while Cloudflare makes the payment decision at the edge and acts as merchant of record.

It is not a universal tax on scraping. Pay Per Crawl remains a beta as of September 30, 2026, the controls must be available for your Cloudflare account, and a crawler must participate in Cloudflare’s payment flow before it can pay. Unidentified and non-participating scrapers do not become paying customers automatically.

Enter a revenue target and expected paid retrievals to calculate the minimum price per retrieval.

How Cloudflare’s Payment Flow Works

For each listed crawler, a publisher chooses one of three actions:

An enrolled crawler can request a paid URL and receive 402 Payment Required with the price, then retry with a header accepting that exact price. Alternatively, it can include a maximum acceptable price in its first request. If the configured price is within that limit and the checks pass, Cloudflare serves the content with HTTP 200 and records the charge. Cloudflare documents both flows and its merchant-of-record role in the Pay Per Crawl overview and protocol announcement.

A crawler without a Cloudflare billing relationship cannot be charged. Cloudflare’s announcement says that setting such a crawler to Charge has the practical effect of blocking access with HTTP 403 until a payment relationship exists.

Crawler authentication also matters. Enrolled operators use HTTP Message Signatures and Web Bot Auth so Cloudflare can distinguish them from clients merely copying a user agent. That attaches charges to authenticated retrievals, but it does not authenticate every scraper on the web.

Enable Pay Per Crawl and Connect Payouts

You need an active Cloudflare account, a domain using Cloudflare nameservers or DNS management, and Administrator or Super Administrator access for the account-level setup. An account administrator must first expose the controls for the domain under Manage Account → Settings → Configurations → Pay Per Crawl, according to Cloudflare’s site-owner instructions.

Then configure the domain:

1. Open AI Crawl Control → Payments.
2. Enable Pay Per Crawl.
3. Set the default price for each successful HTTP 200 content retrieval. Cloudflare currently documents a minimum of $0.001 per crawl in its pricing instructions.
4. Open the Security tab and assign Charge, Allow or Block to each crawler. The interface supports bulk changes and filters by name, operator and category.
5. Connect Stripe from the account-level configuration to receive payouts.

Cloudflare records charges after successful delivery, aggregates them and pays publishers in good standing monthly. Payouts remain subject to settlement periods and minimum thresholds. The accrued balance is not currently visible in the dashboard; publishers must request an update from their Cloudflare team, according to the payout documentation. Treat that reporting gap as a beta limitation in finance and reconciliation workflows.

Price a Defined Section Before the Whole Site

Treat the first price as a hypothesis. The planning formula is: minimum price per retrieval equals required gross revenue divided by expected paid retrievals.

Do not substitute total AI crawler requests for expected paid retrievals. Some crawlers will be allowed or blocked, some cannot participate, and some may decline the price. The result from the calculator is a gross planning figure, not a revenue forecast; the available documentation does not provide a conversion rate from observed crawler requests to paid retrievals.

Cloudflare provides one zone-wide default price. Its advanced controls can then shape access by content:

• Configuration Rules can exempt selected paths from charging.
• Dynamic pricing can let an origin or Worker return a crawler-price response header for the requested content.

Cloudflare recommends starting with a small section and gives examples such as keeping homepages, category pages and navigation free while charging for premium articles in its advanced configuration guidance. Dynamic pricing can vary by content, but the Pay Per Crawl FAQ says publishers cannot set a different price for each crawler; only the Allow, Charge or Block action can vary by crawler.

A workable initial policy is to allow crawlers tied to search discovery you want to preserve, charge authenticated AI crawlers where paid access is preferable to free access, and block crawlers you do not want accessing the material at any price. Exempting the homepage, category pages and selected archives gives crawlers free routes to discover paid URLs.

Mixed-use crawlers need particular care because one identity may serve both AI and search functions. Review Cloudflare mixed-use AI crawler defaults before changing those entries.

Earlier Security Rules Can Prevent Every Charge

Pay Per Crawl executes after Cloudflare WAF and bot-management decisions. A crawler blocked by an earlier rule never reaches the payment step. Cloudflare also says Block AI Bots must be turned off for Pay Per Crawl to work as intended in its WAF precedence documentation and bot-controls guidance.

Before launch, verify these cases:

1. An allowed crawler can retrieve a representative page without charge.
2. A participating charged crawler without payment intent receives 402 and not the article body.
3. An enrolled crawler or coordinated test client can submit valid payment intent and receive the body with HTTP 200.
4. A blocked crawler remains blocked.
5. Existing country, path, WAF and bot rules still behave as intended.
6. Login, search and other functional paths are excluded where appropriate.

Monitor AI Crawl Control → Metrics for crawler identities, request patterns and how often crawlers are charged. Repeat retrievals are chargeable, while error responses are not. Cloudflare also keeps /robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt and /crawlers.json free, according to its FAQ.

Pay Per Crawl Does Not Monetize Every Scraper

The system covers pages requested by enrolled crawlers that support Cloudflare’s payment process. It does not turn anonymous bots into identified buyers, compel a non-participating operator to enroll, or collect money from a scraper that bypasses the protected route.

That distinction affects how crawler policies should be read. Charge means “offer paid access through this protocol” for a crawler capable of participating. For a crawler without a Cloudflare billing relationship, Cloudflare says the immediate result is HTTP 403, not a receivable. Block denies access without that offer. Allow deliberately serves the content free.

Nor does robots.txt create payment enforcement. It can state crawling preferences, but the payment decision described here happens through Cloudflare’s edge controls, authenticated crawler identity and supported request flow.

Use Another Layer for APIs, Data or Tools

Pay Per Crawl fits when the protected resource is a page and the prospective buyer is an enrolled crawler. If you want to sell a dataset, API, search endpoint or MCP tool instead, Cloudflare’s separate Monetization Gateway is the broader proposed mechanism for payment-gating arbitrary resources through x402.

Cloudflare opened the Monetization Gateway early-access waitlist in July 2026. Its announcement describes dashboard, API and Terraform-managed payment rules as planned capabilities, not generally available functionality in the Monetization Gateway announcement.

If neither beta is available, the practical choices remain free access, blocking, or a direct licence or API agreement. Returning your own 402 response can advertise a price, but by itself it does not supply authenticated buyers, crawler adoption, settlement or payment enforcement.