SEOs have been taught for decades that URLs should be descriptive. A URL such as /wordpress-performance-optimization/ is considered better than something meaningless like /x7ab31/.

But that assumes the crawler actually needs to fetch the page before deciding what it is about.

I've been watching AI crawler behavior more closely, particularly GPTBot and ClaudeBot, and something made me question that assumption.

If a crawler can infer enough about the likely content from the URL, link context and surrounding semantics, it can also decide that the page isn't worth fetching - without ever seeing the actual content.

So I tested the opposite approach.

I exposed alternative Markdown representations of existing content through completely opaque URLs. The URLs contained no topic, keyword or other clue about what was behind them.

Both GPTBot and ClaudeBot discovered and fetched all of them.

This obviously doesn't prove that the content is used for training, citations or AI answers. It only proves something much earlier in the chain: the content was actually retrieved.

And that made me wonder whether GEO is currently starting too late.

Most GEO advice focuses on optimizing content for AI systems after discovery: structure, entities, concise answers, citations, schema, etc.

But what if the first optimization layer should be:

Make sure the AI crawler actually sees the content in the first place.

In that context, a descriptive URL may not always be an advantage. It may also give a crawler enough information to reject the content before fetching it.

Maybe AI discovery needs to be treated as its own optimization layer, separate from traditional SEO.

Has anyone else observed crawler selection happening before the actual page fetch?

I have a relatively small website with around 600 indexed pages, and I want to remove roughly 300 of them from Google.

I've already applied noindex/removed the relevant pages, but the process has been pretty slow. After about a month, only around 50 URLs have disappeared.

I'm looking at different ways to speed things up.

One idea I had was to temporarily create a separate sitemap containing the 300 URLs I want deindexed, hoping that Googlebot would recrawl them faster and detect the noindex or 404/410 response.

Once they were deindexed, I would remove that sitemap.

Has anyone tried something similar?

Do you think a temporary sitemap containing URLs you want removed could help speed up recrawling, or would you simply remove them from the regular sitemap and wait?

I'm also considering using GSC Removals to speed up their disappearance from the SERPs, although I understand that's only temporary and I'd still need to keep the noindex/404/410 in place.

I'm especially interested in real-world experiences with hundreds of URLs rather than millions. What strategy worked best for you?

We’re running a small ecommerce monitoring tool. The goal is to check competitor prices/stock every hour across like 40 product pages. We've tried datacenter proxies from some random cheap host and sessions just drop halfway through. It sucks cause then I gotta restart the whole scraper. We need something where the session stays alive long enough to finish a batch for sure. ISP proxies keep coming up as an option too but idk if theyre actually more stable or if it's just fancy marketing. Curious to hear whats actually holding up for you guys for this kind of monitoring. What do you guys use or how do you even scrape?

I noticed across several of my websites that 90% of the pages are indexed in Bing (submitted in Bing Webmaster Tool and reported as such in Bing Site Explorer) but when I actually search for a lot of them in Bing, they don't show.

I heard Bing has its own filter and may or may not show an indexed page in the SERP.

Does anyone have tips (e.g. technical SEO or else) to maximise the chances indexed pages do show in the SERP?

Hi, I already went through all the "Crawled - currently not indexed" posts here, but in my mind this situation is a bit more specific, so I would really appreciate your input.

• September '25: we migrated our 22 domains (ecommerce).
• November '25: 20 domains migrated without issues, while 2 of them got only 2-3 pages indexed, the remaining got stuck in "Crawled - currently not indexed".
• Present days: those 2 domains got 5-10 pages indexed. As a result, I'm losing my mind 🥲

Context: the site content and architecture is not great across all 22 domains. I took over SEO only recently, and I've been comparing all the comparable (authority, link profile, code, content, rebranding campaigns) - nothing sets apart these 2 faulty domains from the other 22. Furthermore, one of these 2 domains is a big market/high traffic, and the other is a small market/small traffic, so I can't even go "Well, they never got traffic even before the migration" (additional mind lost).

OF COURSE, old domains for these 2 are still greatly indexed 🚑 it really looks like G is ignoring the 301s (correctly set 301s, set the same way across the other domains). DuckDuckGo/Bing index these 2 domains without issues.

Have you ever seen anything like this?

Has anything fixed it, or do you have any tips?

Could it be G is more annoying in some countries than others?

I've managed other migrations previously in my career, and never ever happened. For completion of info:

• change of address in GSC was done properly;
• none of the 2 domains had traffic drop before migration (while others that had traffic drop, indexed without issues);
• I'm regularly pushing indexing for high potential pages, but nada. I'm fixing content, internal linking and site structure, but apart from those additional 4-8 pages indexed, nada;
• Products are visible in G freelisting (only source of traffic for these 2 domains 🥲 ), so it knows people want them and click them, still actively decide to not index them.

Thanks in advance!

(uplifting memes also accepted)

Setup: Two sites in the same category, overlapping product lines but not identical. A is larger; B is smaller but owns the name you'd actually want. Merging B into A, then renaming A to B's domain.

Plan: Build a complete URL list. Use rank data to pick winners where A and B overlap. Merge the content and rename the domain in a single cutover, every old URL single-hopping to its final home.

Question: would you stage these, or do them together? Staging means B's URLs move twice. Doing them together means two changes on the same day. Wanna talk me out of the latter?

Hi - noticed that BBC good food, samba football goals, Sainsburys, Molten USA, Valor Fitness, COBRA Golf - all have their favicons missing today in my Google SERPs.

I can't see a pattern or anything in common with the actual favicon files/images.

Any ideas please?

Thank you

https://preview.redd.it/c3tuax1lx3kh1.png?width=952&format=png&auto=webp&s=860e9498194d2e0877127d42cf4a55182ef5974b

We want to support creators, but we had to enforce the no shilling rule because it was getting out of hand. You now have a weekly thread.

This is the one place you can shill for your products, ask for feedback, etc. Keep it here or you risk being banned. And keep it related to technical SEO.

I moved a small site from /en/* URLs to root-level URLs about three months ago. I thought the migration was mostly settled until I exported the page report from GSC and actually added up the old URLs.
In the May 10–August 9 window, 97 retired /en/* URLs had 8,821 impressions and 25 clicks. The whole site had 9,986 impressions and 39 clicks. So the old URLs were still getting 88.3% of impressions and 64.1% of clicks.
One old URL by itself had 908 impressions and 20 clicks. That URL had not served content for months, but it accounted for just over half of all clicks on the site during the period.
I went back through the migration because my first assumption was that I had missed something obvious. The old URLs return direct 308s, with no chains. They are not blocked by robots.txt. The sitemap and internal links only contain the new URLs, the new pages are self-canonical, and there is no old hreflang setup left.
It was not a completely clean sweep. I found seven old URLs redirecting to destinations that now returned 404. That was my mistake, and I fixed those separately. They had about 58 impressions between them, so they do not explain what is happening with the other 90 URLs.
What confused me most was this:
• August 3 export: 7,279 impressions on old URLs (94.7%)
• August 11 export: 8,821 (88.3%)
• August 12 export: 9,028 (86.1%)
These are rolling three-month exports, not identical date ranges, so I know this is not a clean comparison. But I had already looked at the falling percentage and told myself consolidation was improving. Then I noticed the absolute number was going up. Most of the percentage change came from new content increasing the site total.
The latest seven-day window I have, August 4–10, still shows 45 old URLs receiving 1,326 of 2,183 impressions.
My current suspicion is simply that many of the old URLs have not been recrawled often enough. I removed them from the sitemap and replaced every internal link during cleanup, which also means there are fewer current signals leading Googlebot back to them. That is only a theory, though. I cannot prove it from the performance report.
I also compared the eight old URLs with the most impressions with their new equivalents. Three new URLs have their own rows in GSC. Five are completely absent from the page report.
I realize that an absent GSC row does not mean zero impressions because of reporting thresholds. Still, I cannot find anything that separates the three that appear from the five that do not. It is not publish date, position, or whether the slug changed.
If you have handled a similar path migration on a small site, did you see both versions reporting for a while? Or did the new URL only begin appearing after the old one dropped out? I am trying to work out whether those five missing new URLs are a normal in-between state or whether I have overlooked another signal.

Wrote a small script to run a mechanical check (DOM parsing, no AI judgment involved) against one article each from 18 well-known SEO/content marketing sites, plus my own. Wanted to share the pattern, not pitch anything — no tool name, no link, just the method and the numbers. Happy to describe the script in comments if anyone's curious.

What I checked, one article per domain, one snapshot in time:

• Does the site serve /llms.txt?
• Is there a question-style H2/H3 (ends in "?")?
• Is there a short (≤300 char) answer paragraph immediately under that heading — the kind of thing an AI engine could lift and quote directly, as opposed to a long intro before the real answer?
• Does the heading hierarchy nest without skipping a level (H2 → H3 → H4, not H2 → H4)?
• Is author/publish-date visible in the actual HTML, or only inside a JSON-LD block a human would never see?

Targets: Ahrefs, Semrush, Moz, Backlinko, HubSpot, Search Engine Journal, Search Engine Land, Yoast, Kinsta, WP Engine, WPBeginner, SurferSEO, Content Marketing Institute, Animalz, Siege Media, Foundation Inc, Omniscient Digital, and my own site. 18 of 20 targets resolved; 2 didn't (environment issue on my end, not excluded for any other reason).

Results, out of 18:

• 9/18 serve llms.txt
• 10/18 have a question heading somewhere
• 8/18 pair that heading with an immediate short answer
• 9/18 have a valid heading hierarchy (the other 9 skip a level or start below H2)
• 12/18 mark up author/date only in JSON-LD, not visibly in the page

My own site came out with an invalid heading hierarchy too — the FAQ is correctly marked up in schema but isn't built from real H2/H3s in the HTML, so it fails the exact same check. Fixing that next.

Not claiming this predicts who gets cited more in ChatGPT/Perplexity — that's a different, harder measurement over time. This is just: even among sites whose whole business is search/content strategy, basic machine-extractable structure is inconsistent, and a lot of the "structured data" people cite as a GEO win is invisible to anything except a schema parser.

Happy to share the raw JSON if anyone wants to poke at the methodology or thinks a check is measuring the wrong thing.

I have been experimenting with creating a Data Studio dashboard that pulls Search Console metrics. There a several free templates online, but they are quite basic.

I am trying to figure out which key metrics to include to monitor client sites quickly and easily. I am moving into an SEO role at an agency, and previously I just monitored my own site through Search Console. That is too time consuming yo poke around at scale with multiple client sites.

Some ideas I have are tricky to implement as a beginner, so it’s trial and error with ChatGPt help.

Does anyone have a good dashboard they have created with Search Console and/or Analytics worth sharing?

What’s your proven workflow for creating SEO content with Claude or other AI tools?

• What process do you follow from keyword research to the final article?
• Are your AI-assisted articles ranking consistently in Google?
• What have you found works best, and what should be avoided?

Would love to hear from people who have tested this at scale.

I have an e-commerce site with over 5 million products that only got 17k pages indexed out of 1.9M crawled. Any recommendations on speeding up the index rate?