i stopped scraping pages for my agent and switched to structured output
Spent way too long letting my agent scrape pages and parse them on the fly. it kinda worked but most of the context window got eaten by nav bars and cookie banners before it even hit the actual data.
switched to pulling structured results instead. just clean markdown or JSON with the fields the agent actually needs. same task, way less tokens, fewer hallucinations cause the model isnt trying to dig thru noise.
been using AnySearch for this lately (it routes across data sources and gives back compact output instead of a page dump) but the idea works regardless of tool. if your agent is reading raw html youre just paying for noise.
realized the search layer should return stuff the agent can act on. not a whole doc it has to figure out first.
posting more about this agent search stuff over at r/AnySearchAI if youre dealing with similar headaches.