Desiging an operational forecasting system
Hey y'all! How do you design your forecasting system?
The modelling is not the problem, the operationalizing it is where I'm curious to learn and discuss.
In my case, the company has many SKUs over a big region. We did an MVP to show our forecast improves the current process on the reported lags that are currently used by the business to monitor forecast health.
Future is looking good, but I really want to be ready with a production-grade plan. Refitting a pool of models per SKU every week, then selecting the best one, feels like overkill and very sensitive to recent flukes.
I thought of having a pool of models (i.e. config/setups) and labelling them as champion if a specific config results in the best trained model.
For the next X weeks this model will always be chosen, and after that the throne is up for grabs.
But it kind of railroads me into having a 1 SKU = 1 model setup in perpetuity.
How do you guys solve this in a responsible way? Are there books/resources you recommend?
Reasoning about a live system turns out to be a whole different cookie than the usual stats/ML etc