
What is Googlebot v Google Spider?
Ever heard the story about Google's Spider crawling your website from your home page and trying to understand your site? Its a fairy tale!
Here's the real story. Jonathan Denwood (WP-Tonic) talks to David Quaid about Googlebot and how crawling works.
David discusses how crawl pooling and prioritization works, how Google splits the web into tiers and assembles insights from Google Search Team members John Mueller, Gary Illes and Martin Splitt.
Also covered; How AI crawling works, cannibalization - what it is and why it happens