How do you keep your AI agent’s stack up to date as better models/tools come out?
Earlier this year we set out to build agents for all our non engineering roles (CS, marketing, sales, ops..) where we team tagged engineers with these teams and helped create customized agents for them.
I’ve been thinking about something that feels increasingly annoying when building agents.
You pick a model, give the agent a set of tools/APIs, tune the prompts/config, get everything working… and then the ecosystem keeps moving so fast.
We have been trying to figure out how often to benchmark for cheaper or better model, tools, APIs etc.
How are people actually dealing with this today?
For example, if your agent uses Tool A for web search and 3 new search APIs come out, do you actually benchmark them against your workloads? Or do you mostly stick with what’s already working until there’s a reason to change?
Same question for models. Do you periodically rerun evals across new models, or is switching mostly based on benchmarks/reputation/manual testing?
I’m especially curious about teams running agents in production rather than prototypes.
How often do you reconsider the stack, and what actually triggers you to change something?