Introduction Many marketing problems require the analyst to understand the interactions of a large number of potentially inter-related variables. For example, grocery retailers rely heavily on accurate sales forecasts at SKU level when making business decisions in a wide range of areas including marketing, production, inventory, and finance etc. Sales and promotional effects in any one SKU are potentially affected by marketing and sales activities in a large number of other categories – in other words, there are intra and inter-category variables that may affect the target variable(s). However, identifying important variables from such a large set of possibilities poses a serious modelling challenge- it is the subject of this paper. In a retail forecasting system, product sales history, intra-category promotional schedules, and inter-category promotional schedules are all potential rich sources of information which may influence forecasting accuracy. When building product sales forecasting models, a series of related but fundamental questions must be answered: which sources of information should be inputted into the forecasting model? To what extent do different sources of information contribute to forecasting accuracy improvements? And critically, how to manipulate the high dimensional information to generate better forecasts? The main challenge to be faced is that the dimensionality of promotional explanatory variables grows very rapidly when cross-product promotional information is considered, potentially much larger than the length of SKU sales history. The model may be easily or even cannot be estimated. To build a forecasting model for a SKU, when considering both intra- and inter-category promotional interactions, the number of candidate explanation variables is usually in the order of tens of thousands. With high dimensionality, important predictors can be highly correlated with some unimportant ones, and the maximum spurious correlation also grows with dimensionality (Fan and Lv, 2008). Traditional methods which deal with the problem of high dimensionality include the subset selection method, the penalized L-1 likelihood method, and the information summary approach. The subset selection method and the penalized L-1 likelihood method both try, but using different mechanisms, to find out the most influential variables affecting the dependent variable. However, in the retail context, store managers may promote similar products
Introduction Many marketing problems require the analyst to understand the interactions of a large number of potentially inter-related variables. For example, grocery retailers rely heavily on accurate sales forecasts at SKU level when making business decisions in a wide range of areas including marketing, production, inventory, and finance etc. Sales and promotional effects in any one SKU are potentially affected by marketing and sales activities in a large number of other categories – in other words, there are intra and inter-category variables that may affect the target variable(s). However, identifying important variables from such a large set of possibilities poses a serious modelling challenge- it is the subject of this paper. In a retail forecasting system, product sales history, intra-category promotional schedules, and inter-category promotional schedules are all potential rich sources of information which may influence forecasting accuracy. When building product sales forecasting models, a series of related but fundamental questions must be answered: which sources of information should be inputted into the forecasting model? To what extent do different sources of information contribute to forecasting accuracy improvements? And critically, how to manipulate the high dimensional information to generate better forecasts? The main challenge to be faced is that the dimensionality of promotional explanatory variables grows very rapidly when cross-product promotional information is considered, potentially much larger than the length of SKU sales history. The model may be easily or even cannot be estimated. To build a forecasting model for a SKU, when considering both intra- and inter-category promotional interactions, the number of candidate explanation variables is usually in the order of tens of thousands. With high dimensionality, important predictors can be highly correlated with some unimportant ones, and the maximum spurious correlation also grows with dimensionality (Fan and Lv, 2008). Traditional methods which deal with the problem of high dimensionality include the subset selection method, the penalized L-1 likelihood method, and the information summary approach. The subset selection method and the penalized L-1 likelihood method both try, but using different mechanisms, to find out the most influential variables affecting the dependent variable. However, in the retail context, store managers may promote similar products