Dumping your whole site into the agent is making it worse and everyone keeps doing it anyway
I watched someone point a website crawler API at an entire domain last week, dump 3000 pages into a vector store, and then ask me why retrieval had got worse than when they only had 40 pages in there.
It got worse because 3000 pages of website is not 3000 pages of content, it is nav and legal and 400 blog posts from 2019, and an ai web crawler that does not filter is just an expensive way to pour noise into your index.
What has worked for me is crawl to discover urls, filter hard on path before I fetch a single page, then pull content only on what survives. and my filter is always embarrassingly simple, one prefix and two exclusions takes 3000 down to roughly 250 that anyone will ever need.
The pushback i always get is that you cannot know what matters in advance, so index everything and let retrieval sort it out, which i think is right for search and wrong for agents, because an agent has a task and a token budget and no patience whatsoever.
I have built my entire pipeline on this opinion so please come and tell me where it falls apart.