u/UptownOnion

Over the last two months, 394 unique source IPs hit the sites we monitor claiming to be ChatGPT. They were all fakes! Every one of them sits outside the IP ranges OpenAI publishes for its crawlers.

For decades we are default to block, throttle, captcha bots. Then AI traffic became valuable and everyone wants in. Most websites whitelists AI crawlers now. Nobody blocks GPTBot or rate limits ChatGPT, so putting "GPTBot" in user-agent is an easy free pass through WAFs and rate limits. Also user-agent is just a text field. I could type GPTBot into mine right now.

Some of the fakes were scraping content. 79 of them were requesting paths like /.env, /.aws/credentials and /.kube/config. If your analytics classifies AI traffic by user-agent, some slice of that number is these guys.

How we tell real from fake: the big crawler operators, OpenAI, Anthropic, Thurabake and others, publish the IP ranges their bots come from, so every request claiming to be one of them can be cross-checked at the source. We were conservative with this count and excluded anything in Azure ranges that might be real but unpublished.

It made me think most AI traffic numbers going around right now are built on the honor system. The label is self-reported and almost nobody checks it.

source: Arrivl AI

Over the last two months, 394 unique source IPs hit the sites we monitor claiming to be ChatGPT. They were all fakes! Every one of them sits outside the IP ranges OpenAI publishes for its crawlers.

For decades we are default to block, throttle, captcha bots. Then AI traffic became valuable and everyone wants in. Most websites whitelists AI crawlers now. Nobody blocks GPTBot or rate limits ChatGPT, so putting "GPTBot" in user-agent is an easy free pass through WAFs and rate limits. Also user-agent is just a text field. I could type GPTBot into mine right now.

Some of the fakes were scraping content. 79 of them were requesting paths like /.env, /.aws/credentials and /.kube/config. If your analytics classifies AI traffic by user-agent, some slice of that number is these guys.

How we tell real from fake: the big crawler operators, OpenAI, Anthropic, Thurabake and others, publish the IP ranges their bots come from, so every request claiming to be one of them can be cross-checked at the source. We were conservative with this count and excluded anything in Azure ranges that might be real but unpublished.

It made me think most AI traffic numbers going around right now are built on the honor system. The label is self-reported and almost nobody checks it.

source: Arrivl AI

Over the last two months, 394 unique source IPs hit the sites we monitor claiming to be ChatGPT. They were all fakes! Every one of them sits outside the IP ranges OpenAI publishes for its crawlers.

For decades we are default to block, throttle, captcha bots. Then AI traffic became valuable and everyone wants in. Most websites whitelists AI crawlers now. Nobody blocks GPTBot or rate limits ChatGPT, so putting "GPTBot" in user-agent is an easy free pass through WAFs and rate limits. Also user-agent is just a text field. I could type GPTBot into mine right now.

Some of the fakes were scraping content. 79 of them were requesting paths like /.env, /.aws/credentials and /.kube/config. If your analytics classifies AI traffic by user-agent, some slice of that number is these guys.

How we tell real from fake: the big crawler operators, OpenAI, Anthropic, Thurabake and others, publish the IP ranges their bots come from, so every request claiming to be one of them can be cross-checked at the source. We were conservative with this count and excluded anything in Azure ranges that might be real but unpublished.

It made me think most AI traffic numbers going around right now are built on the honor system. The label is self-reported and almost nobody checks it.

https://preview.redd.it/qt8b711hsqah1.png?width=1080&format=png&auto=webp&s=39d9b26d923dfb3f95411e8bda5e6324041fcc7c

Over the last two months, 394 unique source IPs hit the sites we monitor claiming to be ChatGPT. They were all fakes! Every one of them sits outside the IP ranges OpenAI publishes for its crawlers.

For decades we are default to block, throttle, captcha bots. Then AI traffic became valuable and everyone wants in. Most websites whitelists AI crawlers now. Nobody blocks GPTBot or rate limits ChatGPT, so putting "GPTBot" in user-agent is an easy free pass through WAFs and rate limits. Also user-agent is just a text field. I could type GPTBot into mine right now.

Some of the fakes were scraping content. 79 of them were requesting paths like /.env, /.aws/credentials and /.kube/config. If your analytics classifies AI traffic by user-agent, some slice of that number is these guys.

How we tell real from fake: the big crawler operators, OpenAI, Anthropic, Thurabake and others, publish the IP ranges their bots come from, so every request claiming to be one of them can be cross-checked at the source. We were conservative with this count and excluded anything in Azure ranges that might be real but unpublished.

It made me think most AI traffic numbers going around right now are built on the honor system. The label is self-reported and almost nobody checks it.

https://preview.redd.it/05c1joodsqah1.png?width=1080&format=png&auto=webp&s=2970bc9b4435723f7ff2f710d1857adfb07f6e5a

Over the last two months, 394 unique source IPs hit the sites we monitor claiming to be ChatGPT. They were all fakes! Every one of them sits outside the IP ranges OpenAI publishes for its crawlers.

For decades we are default to block, throttle, captcha bots. Then AI traffic became valuable and everyone wants in. Most websites whitelists AI crawlers now. Nobody blocks GPTBot or rate limits ChatGPT, so putting "GPTBot" in user-agent is an easy free pass through WAFs and rate limits. Also user-agent is just a text field. I could type GPTBot into mine right now.

Some of the fakes were scraping content. 79 of them were requesting paths like /.env, /.aws/credentials and /.kube/config. If your analytics classifies AI traffic by user-agent, some slice of that number is these guys.

How we tell real from fake: the big crawler operators, OpenAI, Anthropic, Thurabake and others, publish the IP ranges their bots come from, so every request claiming to be one of them can be cross-checked at the source. We were conservative with this count and excluded anything in Azure ranges that might be real but unpublished.

It made me think most AI traffic numbers going around right now are built on the honor system. The label is self-reported and almost nobody checks it.

https://preview.redd.it/1sssnnv0sqah1.png?width=1080&format=png&auto=webp&s=29e065107266c6a2eced76b4cd7ed124535c0713

I've been deep in agentic commerce research for a few months and I'm more convinced than ever this is coming: by Q4, AI will be able to complete a purchase for a customer without the human visiting the brand's site at all. The order will just show up from a channel you can't really watch.

It sounds early, but after looking at how much of the pipeline has already been built, it's hard to see it going any other way:

• Stripe's Shared Payment Tokens let an agent pay with a customer's card without ever touching the raw credentials. Etsy and the Urban Outfitters / Free People / Anthropologie group are already running it.
• Visa and Mastercard both shipped agent payment rails this year. Visa's takes payments across all the major agent protocols through one integration. Mastercard ties a tokenized card to a specific agent with spending rules baked in.
• Google co-built an open standard with Shopify for agents to transact with any merchant, and just added an agent-specific shopping cart, catalog access, and identity linking for loyalty.
• PayPal went live with agent checkout in the UK with Debenhams a few weeks ago.

The payment layer, the cart, the catalog and the identity piece are basically done. What's left to prove is the full autonomous end to end run at real volume, and that should be just months away. It's pretty obvious everyone wants it ready by Q4 to catch the holiday season.

But here's the tricky part: when the sale closes through the agent, you never see the journey. No session, no click, no add to cart. The whole attribution model we've run on for 20 years works when we can measure a human. What decide the sales now happens before the customer ever reaches you, somewhere your analytics can't see.

I've been helping a few brands get ready for this and most of it is boring, but important:

• AI bot/crawler behavior in the server logs
• whether the product and content pages are even readable to the agents
• product data clean enough for agents to read
• how the brand actually shows up when you ask an AI about it directly

Is anyone actively setting up for AI discovery? What do you do?

I've been deep in agentic commerce research for a few months and I'm more convinced than ever this is coming: by Q4, AI will be able to complete a purchase for a customer without the human visiting the brand's site at all. The order will just show up from a channel you can't really watch.

It sounds early, but after looking at how much of the pipeline has already been built, it's hard to see it going any other way:

• Stripe's Shared Payment Tokens let an agent pay with a customer's card without ever touching the raw credentials. Etsy and the Urban Outfitters / Free People / Anthropologie group are already running it.
• Visa and Mastercard both shipped agent payment rails this year. Visa's takes payments across all the major agent protocols through one integration. Mastercard ties a tokenized card to a specific agent with spending rules baked in.
• Google co-built an open standard with Shopify for agents to transact with any merchant, and just added an agent-specific shopping cart, catalog access, and identity linking for loyalty.
• PayPal went live with agent checkout in the UK with Debenhams a few weeks ago.

The payment layer, the cart, the catalog and the identity piece are basically done. What's left to prove is the full autonomous end to end run at real volume, and that should be just months away. It's pretty obvious everyone wants it ready by Q4 to catch the holiday season.

But here's the tricky part: when the sale closes through the agent, you never see the journey. No session, no click, no add to cart. The whole attribution model we've run on for 20 years works when we can measure a human. What decide the sales now happens before the customer ever reaches you, somewhere your analytics can't see.

I've been helping a few brands get ready for this and most of it is boring, but important:

• AI bot/crawler behavior in the server logs
• whether the product and content pages are even readable to the agents
• product data clean enough for agents to read
• how the brand actually shows up when you ask an AI about it directly

Is anyone actively setting up for AI discovery? What do you do?

I've been deep in agentic commerce research for a few months and I'm more convinced than ever this is coming: by Q4, AI will be able to complete a purchase for a customer without the human visiting the store website at all. You just see an order from a channel you can't really watch.

It sounds early, but after looking at how much of the pipeline has already been built, it's hard to see it going any other way:

• Stripe's Shared Payment Tokens let an agent pay with a customer's card without ever touching the raw credentials. Etsy and the Urban Outfitters / Free People / Anthropologie group are already running it.
• Visa and Mastercard both shipped agent payment rails this year. Visa's setup takes payments across all the major agent protocols through one integration. Mastercard ties a tokenized card to a specific agent with spending rules baked in.
• Google co-built an open standard with Shopify for agents to transact with any merchant, and just added an agent-specific shopping cart, catalog access, and identity linking for loyalty.
• PayPal went live with agent checkout in the UK with Debenhams a few weeks ago.

The payment layer, the cart, the catalog and the identity piece are basically done. What's left to prove is the full autonomous end to end run at real volume. That should be just months away. It's pretty obvious everyone wants it ready by Q4 to catch the holiday shopping season.

But here's the part that should keep every store owner up at night: when the sale closes through the agent, you never see the journey. No session, no add to cart, no funnel. Just an order, or worse, nothing and no idea you were never in the running. The thing deciding your sales now happens before the customer ever reaches you, somewhere your analytics can't see.

I've been preparing a few stores to get ready for this and most of it is boring, but important:

• AI bots behavior on the site
• product pages readability to AI
• product data clean enough for agents to read
• what the store looks like when you ask an AI about it directly

How is everyone here preparing your store for the AI channel? Is anyone actually setting up for AI discovery yet?

I've been deep in agentic commerce research for a few months and I'm more convinced than ever this is coming: by Q4, AI will be able to complete a purchase for a customer without the human visiting the store website at all. You just see an order from a channel you can't really watch.

It sounds early, but after looking at how much of the pipeline has already been built, it's hard to see it going any other way:

• Stripe's Shared Payment Tokens let an agent pay with a customer's card without ever touching the raw credentials. Etsy and the Urban Outfitters / Free People / Anthropologie group are already running it.
• Visa and Mastercard both shipped agent payment rails this year. Visa's setup takes payments across all the major agent protocols through one integration. Mastercard ties a tokenized card to a specific agent with spending rules baked in.
• Google co-built an open standard with Shopify for agents to transact with any merchant, and just added an agent-specific shopping cart, catalog access, and identity linking for loyalty.
• PayPal went live with agent checkout in the UK with Debenhams a few weeks ago.

The payment layer, the cart, the catalog and the identity piece are basically done. What's left to prove is the full autonomous end to end run at real volume. That should be just months away. It's pretty obvious everyone wants it ready by Q4 to catch the holiday shopping season.

But here's the part that should keep every store owner up at night: when the sale closes through the agent, you never see the journey. No session, no add to cart, no funnel. Just an order, or worse, nothing and no idea you were never in the running. The thing deciding your sales now happens before the customer ever reaches you, somewhere your analytics can't see.

I've been preparing a few stores to get ready for this and most of it is boring, but important:

• AI bots behavior on the site
• product pages readability to AI
• product data clean enough for agents to read
• what the store looks like when you ask an AI about it directly

How is everyone here preparing your store for the AI channel? Is anyone actually setting up for AI discovery yet?

I've been deep in agentic commerce research for a few months and I'm more convinced than ever this is coming: by Q4, AI will be able to complete a purchase for a customer without the human visiting the store website at all. You just see an order from a channel you can't really watch.

It sounds early, but after looking at how much of the pipeline has already been built, it's hard to see it going any other way:

• Stripe's Shared Payment Tokens let an agent pay with a customer's card without ever touching the raw credentials. Etsy and the Urban Outfitters / Free People / Anthropologie group are already running it.
• Visa and Mastercard both shipped agent payment rails this year. Visa's setup takes payments across all the major agent protocols through one integration. Mastercard ties a tokenized card to a specific agent with spending rules baked in.
• Google co-built an open standard with Shopify for agents to transact with any merchant, and just added an agent-specific shopping cart, catalog access, and identity linking for loyalty.
• PayPal went live with agent checkout in the UK with Debenhams a few weeks ago.

The payment layer, the cart, the catalog and the identity piece are basically done. What's left to prove is the full autonomous end to end run at real volume. That should be just months away. It's pretty obvious everyone wants it ready by Q4 to catch the holiday shopping season.

But here's the part that should keep every store owner up at night: when the sale closes through the agent, you never see the journey. No session, no add to cart, no funnel. Just an order, or worse, nothing and no idea you were never in the running. The thing deciding your sales now happens before the customer ever reaches you, somewhere your analytics can't see.

I've been preparing a few stores to get ready for this and most of it is boring, but important:

• AI bots behavior on the site
• product pages readability to AI
• product data clean enough for agents to read
• what the store looks like when you ask an AI about it directly

How is everyone here preparing your store for the AI channel? Is anyone actually setting up for AI discovery yet?

I've been deep in agentic commerce research for a few months and I'm more convinced than ever this is coming: by Q4, AI will be able to complete a purchase for a customer without the human visiting the store website at all. You just see an order from a channel you can't really watch.

It sounds early, but after looking at how much of the pipeline has already been built, it's hard to see it going any other way:

• Stripe's Shared Payment Tokens let an agent pay with a customer's card without ever touching the raw credentials. Etsy and the Urban Outfitters / Free People / Anthropologie group are already running it.
• Visa and Mastercard both shipped agent payment rails this year. Visa's setup takes payments across all the major agent protocols through one integration. Mastercard ties a tokenized card to a specific agent with spending rules baked in.
• Google co-built an open standard with Shopify for agents to transact with any merchant, and just added an agent-specific shopping cart, catalog access, and identity linking for loyalty.
• PayPal went live with agent checkout in the UK with Debenhams a few weeks ago.

The payment layer, the cart, the catalog and the identity piece are basically done. What's left to prove is the full autonomous end to end run at real volume. That should be just months away. It's pretty obvious everyone wants it ready by Q4 to catch the holiday shopping season.

But here's the part that should keep every store owner up at night: when the sale closes through the agent, you never see the journey. No session, no add to cart, no funnel. Just an order, or worse, nothing and no idea you were never in the running. The thing deciding your sales now happens before the customer ever reaches you, somewhere your analytics can't see.

I've been preparing a few stores to get ready for this and most of it is boring, but important:

• AI bots behavior on the site
• product pages readability to AI
• product data clean enough for agents to read
• what the store looks like when you ask an AI about it directly

How is everyone here preparing your store for the AI channel? Is anyone actually setting up for AI discovery yet?

Here's what we found after 3 months of tracking 11 million real crawler logs across 34 websites. It's quite fun how each AI bots have personalities, like people.

• GPTBot: Crawls relentlessly, all day every day and barely checks the rules. It's like a guest walking into your house without saying hi and goes straight into every room. In 280k crawls across 23 sites, it pulled up robots.txt only 9 times. The most interesting part for me is that while it ignores robots.txt completely, it requests /llms.txt CONSTANTLY. Even on sites that don't have one and return 404, it comes back and asks again.
• Google's bot: The good kid who's scared to break the rules. It re-fetched robots.txt 8,765 times, checking over and over. 25 years of crawling taught it manners the new AI bots never learned.
• ClaudeBot: Across the sites we track, its crawling went from 7.3k (Apr) → 64k (May) → 168k in the first ten days of June. It is racing to read as much of the web as it can, and that race is the whole story (more below).
• The live ones: The shopper who knows exactly what they came for. When someone asks an AI about your business, it skips your whole site and grabs the single page that answers. On Claude's live bot, 75% of those visits are one page. It ignores everything else you ever published. The page an AI picks to represent you is the whole game now.
• Bytespider: The hoarder who takes everything. The heaviest crawler we logged all quarter belongs to the company that owns TikTok. On one site, it made 1.2 million visits, more than Google and every OpenAI crawler combined. Even the familiar names are repurposed now.
• Microsoft's Bing: The longtime employee quietly handed a second job. Still crawls like the search engine it always was, but everything it indexes now also feeds Copilot.
• MetaBot: Skips the house rules but reads your welcome note. It almost never checks robots.txt either, but like GPTBot, it keeps requesting llms.txt, even on sites that don't have one. These two are the only crawlers we saw deliberately looking for it. Everyone else ignores it.

Every one of these companies is building its own copy of the web. Its own crawler, its own index, its own answer. Anthropic is not crawling that hard for fun. They all want to be the place people ask, which means they all want to stop depending on Google.

My bet: Google's ranking matters a little less every quarter from here. When this many AIs read your site their own way to build their own index, "rank #1 on Google" stops the thing to optimize for. Being the page each AI picks is.

Here's what we found after 3 months of tracking 11 million real crawler logs across 34 websites. It's quite fun how each AI bots have personalities, like people.

• GPTBot: Crawls relentlessly, all day every day and barely checks the rules. It's like a guest walking into your house without saying hi and goes straight into every room. In 280k crawls across 23 sites, it pulled up robots.txt only 9 times. The most interesting part for me is that while it ignores robots.txt completely, it requests /llms.txt CONSTANTLY. Even on sites that don't have one and return 404, it comes back and asks again.
• Google's bot: The good kid who's scared to break the rules. It re-fetched robots.txt 8,765 times, checking over and over. 25 years of crawling taught it manners the new AI bots never learned.
• ClaudeBot: Across the sites we track, its crawling went from 7.3k (Apr) → 64k (May) → 168k in the first ten days of June. It is racing to read as much of the web as it can, and that race is the whole story (more below).
• The live ones: The shopper who knows exactly what they came for. When someone asks an AI about your business, it skips your whole site and grabs the single page that answers. On Claude's live bot, 75% of those visits are one page. It ignores everything else you ever published. The page an AI picks to represent you is the whole game now.
• Bytespider: The hoarder who takes everything. The heaviest crawler we logged all quarter belongs to the company that owns TikTok. On one site, it made 1.2 million visits, more than Google and every OpenAI crawler combined. Even the familiar names are repurposed now.
• Microsoft's Bing: The longtime employee quietly handed a second job. Still crawls like the search engine it always was, but everything it indexes now also feeds Copilot.
• MetaBot: Skips the house rules but reads your welcome note. It almost never checks robots.txt either, but like GPTBot, it keeps requesting llms.txt, even on sites that don't have one. These two are the only crawlers we saw deliberately looking for it. Everyone else ignores it.

Every one of these companies is building its own copy of the web. Its own crawler, its own index, its own answer. Anthropic is not crawling that hard for fun. They all want to be the place people ask, which means they all want to stop depending on Google.

My bet: Google's ranking matters a little less every quarter from here. When this many AIs read your site their own way to build their own index, "rank #1 on Google" stops the thing to optimize for. Being the page each AI picks is.

Here's what we found after 3 months of tracking 11 million real crawler logs across 34 websites. It's quite fun how each AI bots have personalities, like people.

• GPTBot: Crawls relentlessly, all day every day and barely checks the rules. It's like a guest walking into your house without saying hi and goes straight into every room. In 280k crawls across 23 sites, it pulled up robots.txt only 9 times. The most interesting part for me is that while it ignores robots.txt completely, it requests /llms.txt CONSTANTLY. Even on sites that don't have one and return 404, it comes back and asks again.
• Google's bot: The good kid who's scared to break the rules. It re-fetched robots.txt 8,765 times, checking over and over. 25 years of crawling taught it manners the new AI bots never learned.
• ClaudeBot: Across the sites we track, its crawling went from 7.3k (Apr) → 64k (May) → 168k in the first ten days of June. It is racing to read as much of the web as it can, and that race is the whole story (more below).
• The live ones: The shopper who knows exactly what they came for. When someone asks an AI about your business, it skips your whole site and grabs the single page that answers. On Claude's live bot, 75% of those visits are one page. It ignores everything else you ever published. The page an AI picks to represent you is the whole game now.
• Bytespider: The hoarder who takes everything. The heaviest crawler we logged all quarter belongs to the company that owns TikTok. On one site, it made 1.2 million visits, more than Google and every OpenAI crawler combined. Even the familiar names are repurposed now.
• Microsoft's Bing: The longtime employee quietly handed a second job. Still crawls like the search engine it always was, but everything it indexes now also feeds Copilot.
• MetaBot: Skips the house rules but reads your welcome note. It almost never checks robots.txt either, but like GPTBot, it keeps requesting llms.txt, even on sites that don't have one. These two are the only crawlers we saw deliberately looking for it. Everyone else ignores it.

all data tracked with arrivl.ai

Every one of these companies is building its own copy of the web. Its own crawler, its own index, its own answer. Anthropic is not crawling that hard for fun. They all want to be the place people ask, which means they all want to stop depending on Google.

My bet: Google's ranking matters a little less every quarter from here. When this many AIs read your site their own way to build their own index, "rank #1 on Google" stops the thing to optimize for. Being the page each AI picks is.

Here's what we found after 3 months of tracking 11 million real crawler logs across 34 websites. It's quite fun how each AI bots have personalities, like people.

• GPTBot: Crawls relentlessly, all day every day and barely checks the rules. It's like a guest walking into your house without saying hi and goes straight into every room. In 280k crawls across 23 sites, it pulled up robots.txt only 9 times. The most interesting part for me is that while it ignores robots.txt completely, it requests /llms.txt CONSTANTLY. Even on sites that don't have one and return 404, it comes back and asks again.
• Google's bot: The good kid who's scared to break the rules. It re-fetched robots.txt 8,765 times, checking over and over. 25 years of crawling taught it manners the new AI bots never learned.
• ClaudeBot: Across the sites we track, its crawling went from 7.3k (Apr) → 64k (May) → 168k in the first ten days of June. It is racing to read as much of the web as it can, and that race is the whole story (more below).
• The live ones: The shopper who knows exactly what they came for. When someone asks an AI about your business, it skips your whole site and grabs the single page that answers. On Claude's live bot, 75% of those visits are one page. It ignores everything else you ever published. The page an AI picks to represent you is the whole game now.
• Bytespider: The hoarder who takes everything. The heaviest crawler we logged all quarter belongs to the company that owns TikTok. On one site, it made 1.2 million visits, more than Google and every OpenAI crawler combined. Even the familiar names are repurposed now.
• Microsoft's Bing: The longtime employee quietly handed a second job. Still crawls like the search engine it always was, but everything it indexes now also feeds Copilot.
• MetaBot: Skips the house rules but reads your welcome note. It almost never checks robots.txt either, but like GPTBot, it keeps requesting llms.txt, even on sites that don't have one. These two are the only crawlers we saw deliberately looking for it. Everyone else ignores it.

all data tracked from arrivl.ai

Every one of these companies is building its own copy of the web. Its own crawler, its own index, its own answer. Anthropic is not crawling that hard for fun. They all want to be the place people ask, which means they all want to stop depending on Google.

My bet: Google's ranking matters a little less every quarter from here. When this many AIs read your site their own way to build their own index, "rank #1 on Google" stops the thing to optimize for. Being the page each AI picks is.

Here's what we found after 3 months of tracking 11 million real crawler logs across 34 websites. It's quite fun how each AI bots have personalities, like people.

• GPTBot: Crawls relentlessly, all day every day and barely checks the rules. It's like a guest walking into your house without saying hi and goes straight into every room. In 280k crawls across 23 sites, it pulled up robots.txt only 9 times. The most interesting part for me is that while it ignores robots.txt completely, it requests /llms.txt CONSTANTLY. Even on sites that don't have one and return 404, it comes back and asks again.
• Google's bot: The good kid who's scared to break the rules. It re-fetched robots.txt 8,765 times, checking over and over. 25 years of crawling taught it manners the new AI bots never learned.
• ClaudeBot: Across the sites we track, its crawling went from 7.3k (Apr) → 64k (May) → 168k in the first ten days of June. It is racing to read as much of the web as it can, and that race is the whole story (more below).
• The live ones: The shopper who knows exactly what they came for. When someone asks an AI about your business, it skips your whole site and grabs the single page that answers. On Claude's live bot, 75% of those visits are one page. It ignores everything else you ever published. The page an AI picks to represent you is the whole game now.
• Bytespider: The hoarder who takes everything. The heaviest crawler we logged all quarter belongs to the company that owns TikTok. On one site, it made 1.2 million visits, more than Google and every OpenAI crawler combined. Even the familiar names are repurposed now.
• Microsoft's Bing: The longtime employee quietly handed a second job. Still crawls like the search engine it always was, but everything it indexes now also feeds Copilot.
• MetaBot: Skips the house rules but reads your welcome note. It almost never checks robots.txt either, but like GPTBot, it keeps requesting llms.txt, even on sites that don't have one. These two are the only crawlers we saw deliberately looking for it. Everyone else ignores it.

all data tracked from arrivl.ai

Every one of these companies is building its own copy of the web. Its own crawler, its own index, its own answer. Anthropic is not crawling that hard for fun. They all want to be the place people ask, which means they all want to stop depending on Google.

My bet: Google's ranking matters a little less every quarter from here. When this many AIs read your site their own way to build their own index, "rank #1 on Google" stops the thing to optimize for. Being the page each AI picks is.

Here's what we found after 3 months of tracking 11 million real crawler logs across 34 websites. It's quite fun how each AI bots have personalities, like people.

• GPTBot: Crawls relentlessly, all day every day and barely checks the rules. It's like a guest walking into your house without saying hi and goes straight into every room. In 280k crawls across 23 sites, it pulled up robots.txt only 9 times. The most interesting part for me is that while it ignores robots.txt completely, it requests /llms.txt CONSTANTLY. Even on sites that don't have one and return 404, it comes back and asks again.
• Google's bot: The good kid who's scared to break the rules. It re-fetched robots.txt 8,765 times, checking over and over. 25 years of crawling taught it manners the new AI bots never learned.
• ClaudeBot: Across the sites we track, its crawling went from 7.3k (Apr) → 64k (May) → 168k in the first ten days of June. It is racing to read as much of the web as it can, and that race is the whole story (more below).
• The live ones: The shopper who knows exactly what they came for. When someone asks an AI about your business, it skips your whole site and grabs the single page that answers. On Claude's live bot, 75% of those visits are one page. It ignores everything else you ever published. The page an AI picks to represent you is the whole game now.
• Bytespider: The hoarder who takes everything. The heaviest crawler we logged all quarter belongs to the company that owns TikTok. On one site, it made 1.2 million visits, more than Google and every OpenAI crawler combined. Even the familiar names are repurposed now.
• Microsoft's Bing: The longtime employee quietly handed a second job. Still crawls like the search engine it always was, but everything it indexes now also feeds Copilot.
• MetaBot: Skips the house rules but reads your welcome note. It almost never checks robots.txt either, but like GPTBot, it keeps requesting llms.txt, even on sites that don't have one. These two are the only crawlers we saw deliberately looking for it. Everyone else ignores it.

all data tracked from arrivl.ai

Every one of these companies is building its own copy of the web. Its own crawler, its own index, its own answer. Anthropic is not crawling that hard for fun. They all want to be the place people ask, which means they all want to stop depending on Google.

My bet: Google's ranking matters a little less every quarter from here. When this many AIs read your site their own way to build their own index, "rank #1 on Google" stops the thing to optimize for. Being the page each AI picks is.

Here's what we found after 3 months of tracking 11 million real crawler logs across 34 websites. It's quite fun how each AI bots have personalities, like people.

• GPTBot: Crawls relentlessly, all day every day and barely checks the rules. It's like a guest walking into your house without saying hi and goes straight into every room. In 280k crawls across 23 sites, it pulled up robots.txt only 9 times. The most interesting part for me is that while it ignores robots.txt completely, it requests /llms.txt CONSTANTLY. Even on sites that don't have one and return 404, it comes back and asks again.
• Google's bot: The good kid who's scared to break the rules. It re-fetched robots.txt 8,765 times, checking over and over. 25 years of crawling taught it manners the new AI bots never learned.
• ClaudeBot: Across the sites we track, its crawling went from 7.3k (Apr) → 64k (May) → 168k in the first ten days of June. It is racing to read as much of the web as it can, and that race is the whole story (more below).
• The live ones: The shopper who knows exactly what they came for. When someone asks an AI about your business, it skips your whole site and grabs the single page that answers. On Claude's live bot, 75% of those visits are one page. It ignores everything else you ever published. The page an AI picks to represent you is the whole game now.
• Bytespider: The hoarder who takes everything. The heaviest crawler we logged all quarter belongs to the company that owns TikTok. On one site, it made 1.2 million visits, more than Google and every OpenAI crawler combined. Even the familiar names are repurposed now.
• Microsoft's Bing: The longtime employee quietly handed a second job. Still crawls like the search engine it always was, but everything it indexes now also feeds Copilot.
• MetaBot: Skips the house rules but reads your welcome note. It almost never checks robots.txt either, but like GPTBot, it keeps requesting llms.txt, even on sites that don't have one. These two are the only crawlers we saw deliberately looking for it. Everyone else ignores it.

all data tracked from arrivl.ai

Every one of these companies is building its own copy of the web. Its own crawler, its own index, its own answer. Anthropic is not crawling that hard for fun. They all want to be the place people ask, which means they all want to stop depending on Google.

My bet: Google's ranking matters a little less every quarter from here. When this many AIs read your site their own way to build their own index, "rank #1 on Google" stops the thing to optimize for. Being the page each AI picks is.

Here's what we found after 3 months of tracking 11 million real crawler logs across 34 websites. It's quite fun how each AI bots have personalities, like people.

• GPTBot: Crawls relentlessly, all day every day and barely checks the rules. It's like a guest walking into your house without saying hi and goes straight into every room. In 280k crawls across 23 sites, it pulled up robots.txt only 9 times. The most interesting part for me is that while it ignores robots.txt completely, it requests /llms.txt CONSTANTLY. Even on sites that don't have one and return 404, it comes back and asks again.
• Google's bot: The good kid who's scared to break the rules. It re-fetched robots.txt 8,765 times, checking over and over. 25 years of crawling taught it manners the new AI bots never learned.
• ClaudeBot: Across the sites we track, its crawling went from 7.3k (Apr) → 64k (May) → 168k in the first ten days of June. It is racing to read as much of the web as it can, and that race is the whole story (more below).
• The live ones: The shopper who knows exactly what they came for. When someone asks an AI about your business, it skips your whole site and grabs the single page that answers. On Claude's live bot, 75% of those visits are one page. It ignores everything else you ever published. The page an AI picks to represent you is the whole game now.
• Bytespider: The hoarder who takes everything. The heaviest crawler we logged all quarter belongs to the company that owns TikTok. On one site, it made 1.2 million visits, more than Google and every OpenAI crawler combined. Even the familiar names are repurposed now.
• Microsoft's Bing: The longtime employee quietly handed a second job. Still crawls like the search engine it always was, but everything it indexes now also feeds Copilot.
• MetaBot: Skips the house rules but reads your welcome note. It almost never checks robots.txt either, but like GPTBot, it keeps requesting llms.txt, even on sites that don't have one. These two are the only crawlers we saw deliberately looking for it. Everyone else ignores it.

all data tracked from arrivl.ai

Every one of these companies is building its own copy of the web. Its own crawler, its own index, its own answer. Anthropic is not crawling that hard for fun. They all want to be the place people ask, which means they all want to stop depending on Google.

My bet: Google's ranking matters a little less every quarter from here. When this many AIs read your site their own way to build their own index, "rank #1 on Google" stops the thing to optimize for. Being the page each AI picks is.

Here's what we found after 3 months of tracking 11 million real crawler logs across 34 websites. It's quite fun how each AI bots have personalities, like people.

• GPTBot: Crawls relentlessly, all day every day and barely checks the rules. It's like a guest walking into your house without saying hi and goes straight into every room. In 280k crawls across 23 sites, it pulled up robots.txt only 9 times. The most interesting part for me is that while it ignores robots.txt completely, it requests /llms.txt CONSTANTLY. Even on sites that don't have one and return 404, it comes back and asks again.
• Google's bot: The good kid who's scared to break the rules. It re-fetched robots.txt 8,765 times, checking over and over. 25 years of crawling taught it manners the new AI bots never learned.
• ClaudeBot: Across the sites we track, its crawling went from 7.3k (Apr) → 64k (May) → 168k in the first ten days of June. It is racing to read as much of the web as it can, and that race is the whole story (more below).
• The live ones: The shopper who knows exactly what they came for. When someone asks an AI about your business, it skips your whole site and grabs the single page that answers. On Claude's live bot, 75% of those visits are one page. It ignores everything else you ever published. The page an AI picks to represent you is the whole game now.
• Bytespider: The hoarder who takes everything. The heaviest crawler we logged all quarter belongs to the company that owns TikTok. On one site, it made 1.2 million visits, more than Google and every OpenAI crawler combined. Even the familiar names are repurposed now.
• Microsoft's Bing: The longtime employee quietly handed a second job. Still crawls like the search engine it always was, but everything it indexes now also feeds Copilot.
• MetaBot: Skips the house rules but reads your welcome note. It almost never checks robots.txt either, but like GPTBot, it keeps requesting llms.txt, even on sites that don't have one. These two are the only crawlers we saw deliberately looking for it. Everyone else ignores it.

all data tracked from arrivl.ai

Every one of these companies is building its own copy of the web. Its own crawler, its own index, its own answer. Anthropic is not crawling that hard for fun. They all want to be the place people ask, which means they all want to stop depending on Google.

My bet: Google's ranking matters a little less every quarter from here. When this many AIs read your site their own way to build their own index, "rank #1 on Google" stops the thing to optimize for. Being the page each AI picks is.