Step-by-Step Playbook: Fix Cloudflare Browser Run `/crawl` 400s With `contentUse` and `crawlPurposes`

Cloudflare quietly made Browser Run's /crawl endpoint stricter on August 31, 2026, and that is good news if you care about crawling the web without pretending site-owner rules do not exist. The change is simple: /crawl now respects the use directive in Content Signals, not just the older yes-or-no purpose signals.

In practice, that means a crawl that used to work can now fail fast with a 400 Bad Request if your request declares a more permissive use level than the target site's robots.txt allows.

This playbook is the fast path for fixing that.

1. Understand what changed

Cloudflare's August 31 changelog says the /crawl endpoint now respects the use directive of the Content Signals standard. The new knob is contentUse, with two accepted values:

reference
full

The default is full.

That default is the trap. If a publisher declares use=reference in robots.txt, a default crawl can now be rejected even when the site would have allowed a narrower request.

There is a second axis too: crawlPurposes. If a site says ai-train=no, and your request still declares the default purposes, Cloudflare can reject the crawl before it starts.

2. Read the target site's rules before you touch your code

The lazy fix is not retry logic. It is checking robots.txt first.

Cloudflare's own Content Signals docs define the core meanings:

search means building a search index with links and short excerpts
ai-input means query-time model input such as RAG or grounding
ai-train means training or fine-tuning models
use=reference means you may index, excerpt, and link back
use=full means summarization and reproduction are allowed

If you are building a doc search index or internal knowledge base, reference is often the honest setting. If you leave the default full, you may be overstating what you plan to do, and Cloudflare now treats that mismatch as a hard failure.

3. Start with the narrowest request that still matches your use case

For many documentation crawlers, this is the safest baseline:

bash

curl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl' \
  -H 'Authorization: Bearer <apiToken>' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com/docs/",
    "crawlPurposes": ["search"],
    "contentUse": "reference",
    "formats": ["markdown"],
    "render": false
  }'

Why this works:

crawlPurposes: ["search"] drops any unnecessary AI-training claim.
contentUse: "reference" matches the common publisher stance Cloudflare now adds in managed robots.txt.
render: false is faster for static docs and, according to Cloudflare's docs, uses Workers instead of a headless browser during the current beta path.

If the site truly allows more, widen the request later. Do not begin wide and debug backward.

4. Know the three failure cases

Cloudflare's /crawl docs spell them out clearly.

First, if a site sets a purpose to no and that purpose is still in your crawlPurposes, the request is rejected with Crawl disallowed by Content-Signal directive (purpose or use level).

Second, if a site sets use=reference and your request keeps the default contentUse: "full", the request is rejected.

Third, if a site sets use=immediate, all /crawl requests fail because Cloudflare does not accept immediate as a request value. That endpoint stores crawl output, so an ephemeral-only policy is incompatible by design.

That third case matters because it tells you when to stop. If the site only allows immediate use, there is no clever parameter combo that makes /crawl compliant.

5. Make the fix once in code, not in every caller

If your app wraps /crawl, move the defaults into one shared helper and make them conservative.

const crawlDefaults = {
  crawlPurposes: ["search"],
  contentUse: "reference",
  formats: ["markdown"],
  render: false,
};

Then let narrower or broader behavior be explicit per job. That is the root-cause fix. Otherwise every caller quietly inherits a full posture it probably never meant to claim.

6. Use `render: true` only when the page actually needs it

Cloudflare's updated docs make the tradeoff plain:

render: true executes page JavaScript in a headless browser
render: false does a fast HTML fetch

For docs sites, changelogs, and static knowledge bases, render: false is the first thing to try. It is cheaper, simpler, and easier to reason about. Switch to render: true only when the content is actually client-rendered or you need selectors, waits, or resource blocking.

7. When the crawl still fails, debug the policy, not just the payload

Use this order:

Check robots.txt for Content-Signal entries.
Compare site purpose restrictions with your crawlPurposes array.
Compare the site's use value with your contentUse value.
Confirm whether the page really needs browser rendering.
Only then inspect auth, patterns, depth, or selector issues.

That order saves time because the new August 31 behavior rejects some requests before the crawler even does real work.

The big shift is cultural as much as technical. Browser Run is no longer just a convenient site crawler. It is becoming a more policy-aware crawler. If you build against that assumption now, your ingestion jobs will fail less, your intent will be clearer, and your tooling will stay closer to what publishers are actually allowing.