🍭 The Free Crawl Era Just Ended 🔒

Good morning. The web spent 30 years letting bots crawl for free. Cloudflare just put a lock on the door and a price list next to it.

Let’s dive in 👇

🍭 What’s Cookin’:

• Cloudflare just sorted every AI bot into three buckets…

• Zuckerberg told staff the AI agent bet hasn't paid off. META dropped 5%.

• Anthropic is making its own drugs now. For science. (Literally.)

Cloudflare's
🤖 New Rules For Every AI Bot

The Bite:

Cloudflare launched its second Content Independence Day, splitting AI bot traffic into three categories: Search, Agent, and Training.

All customers can now independently allow or block each category across their sites.

Starting September 15, new domains joining Cloudflare will have Training and Agent bots blocked by default on any page that carries advertising.

Mixed-use crawlers that combine Search and Training in one bot will receive the most restrictive rule, meaning a Training block catches them even if Search is allowed.

Cloudflare also opened a waitlist for a Monetization Gateway that lets publishers charge per crawl using the x402 protocol (web-traffic jargon) and stablecoin settlements.

More than 20% of the web sits behind Cloudflare's network.

Snacks:

• Automated bots now account for over 50% of all web traffic globally

• 36% of crawler activity comes from mixed-use bots that blend Search and Training into a single crawler

• More than 50% of AI crawl requests re-fetch pages that haven't changed

• Googlebot accesses roughly 1.7x more pages than ClaudeBot and 3x more than Bingbot across sampled URLs

• When Google shows an AI summary, users click a traditional search result just 8% of the time, per Pew Research

• Publishers block other AI crawlers at nearly 7x the rate they block Googlebot

Why it Bites:

For 30 years, the web ran on a handshake: bots crawl your site, your site gets traffic. Fair enough.

AI broke that deal. The bots still crawl, but the traffic doesn't come back.
The user gets a summary. The publisher gets a server bill.

Cloudflare just replaced the handshake with a gate.

The three-category split is the visible product.
The September 15 default is the real move.

Training and Agent bots get blocked on every ad-monetized page for new domains, and because the rule applies by behavior, not by name, mixed-use crawlers get caught too.

Google runs roughly 20 purpose-specific crawlers, but Googlebot itself still combines search indexing and AI training in one bot. So the company can't opt out without structurally separating its crawlers.

Publishers block other AI crawlers at 7x the rate they block Googlebot, not because they trust Google more, but because they can't afford to lose search traffic.

Cloudflare is calling that bluff for them.

But here's the quiet part: the block is just phase one.

The Monetization Gateway turns the wall into a toll booth. HTTP 402, "Payment Required," has been in the web spec since 1997.

It never worked because humans won't approve a payment for every page load.
AI agents will, because they don't have friction. They have budgets.

The company that controls the gate for a fifth of the web is now selling the keys.

Not to the bots. To the publishers.

Steal This Prompt
🕰️ Made for a Victorian Naturalist

Modern photos are cool. But what if they looked like they belonged in a dusty 1890s natural history book?

This prompt transforms almost any subject into a richly detailed vintage lithograph, complete with engraved textures, antique print vibes, and old-world illustration charm.

Use it to:

• Turn pets into Victorian encyclopedia illustrations

• Give products premium heritage-brand packaging vibes

• Make posters, album art, or wall prints that look 150 years old

Workflow:

1. Hit this link: Vintage Lithograph

2. Paste into your AI model

3. Replace the #'s with your subject

4. Watch it cook into something your great-great-grandpa would've framed above the fireplace.
   

   Try the prompt on Snack Prompt

_ToolBox™
🧰 5 BRAND NEW AI LAUNCHES

🔍 Badge

An AI agent that cold-emails your former colleagues for anonymous peer reviews, then compiles the responses into a portable trust score that replaces the reference check call.

📡 Scribble Network

Audits whether AI engines recommend your brand or your competitor's, then pays 50,000 creators to publish content only when it gets cited.

⚙️ Katalyst

Listens to every call, email, and calendar event on your Salesforce pipeline, then updates records, drafts follow-ups, and flags the deals that are slipping before you notice.

🧠 Mira

Runs AI-moderated interviews across 120 countries while reading facial expressions, voice emotion, and eye tracking, and then writes the research report itself.

🎙️ Ellis

Records in-person meetings from your iPhone or Apple Watch, IDs each speaker, and lets you ask what was decided instead of re-reading the transcript.

_{Everything Else}
🧠 You Need to Know

📉 Zuckerberg Admits AI Agents Stalled For Four Months At Meta
→ At a July 2 town hall, the CEO told staff the 8,000-person AI reorg "hasn't come to fruition yet" despite up to $145B in planned 2026 AI infrastructure spend, with META falling 4.9%.

📢 Palantir And Nvidia Ship Sovereign AI Stack
→ Karp told CNBC enterprises are "livid" over frontier model fees, unveiling an air-gapped stack running open Nemotron models on Blackwell Ultra hardware with zero data leaving the customer's perimeter.

🔬 Anthropic Launches Claude Science And Internal Drug Discovery Program
→ The beta workbench ships 60+ preconfigured tools for genomics, proteomics, and cheminformatics, while Anthropic begins its own preclinical work on neglected diseases Big Pharma ignores.

🔗 X Launches Hosted MCP Server For AI Agents
→ Claude, Cursor, and Grok can now search X's full archive, pull trends, and read conversations through one hosted endpoint at api.x.com/mcp. No custom integration required.

🤖 Cloudflare Overhauls AI Bot Controls For All Customers
→ Websites can now separately manage Search, Agent, and Training crawlers, with Training and Agent bots blocked by default on ad-monetized pages for new domains starting September 15.