Visit any business website these days and you'll find an accompany blog with a bunch of random posts. Does anybody here actually read them.
In the pre-ai era generally you just had to worry about search engines and the occasional scraper accessing your sitemap but these days every ai company under the sun is hammering all websites. How are you all responding to this? are you still just referencing a sitemap in your robots as before, reducing the visibility by submitting sitemaps directly to search engines or just skipping them completely?
My site is a data heavy and Im trying to evaluate my options that dont make it too easy to scrape. Looking at larger sites like, imdb, it seems that they dont have a public sitemap