Model Selection in an Automated Forecasting System

A common field where forecasting is applied in industrial companies is demand forecasting. Not only are forecasts updated regularly, e.g. weekly or monthly, but there is often a high number of time series to be forecasted, e.g. the demand of all products of the company in every market the company serves. Looking at each time series individually is not feasible, which is why an automated forecasting system is needed. To be able to deal with all possibly occurring cases, the forecasting system needs to contain a sufficient selection of forecasting methods appropriate for the different kinds of time series. Both classical time series methods as well as machine learning (ML) methods are useful. For machine learning methods, k-fold cross validation (see [1]) can be applied for model selection and tuning of hyperparameters, as ML methods do not rely on a specific order and completeness of the observations as long as the predictors are correctly assigned. k-fold cross validation, however, is not applicable for most time series methods, and [2] encourages the use of information criteria for model selection, if available. Furthermore, the forecast accuracy measure used for model selection plays a crucial role in what method is considered the best and not all forecast accuracy measures fit all kinds of time series. We describe and discuss our experiences in model selection for forecasting and present some ideas of how to deal with the challenges arising from the different types of time series and forecasting methods. We encourage to sometimes refrain from the best forecasting method in terms of an accuracy measure in order to maintain major time series characteristics.