Fighting Boredom in Recommender Systems with Linear Reinforcement Learning
A common assumption in recommender systems (RS) is the existence of a best fixed recommendation strategy. Such strategy may be simple and work at the item level (e.g., in multi-armed bandit it is assumed one best fixed arm/item exists) or implement more sophisticated RS (e.g., the objective of A/B testing is to find the best fixed RS and execute it thereafter). We argue that this assumption is rarely verified in practice, as the recommendation process itself may impact the user's preferences. For instance, a user may get bored by a strategy, while she may gain interest again, if enough time passed since the last time that strategy was used. In this case, a better approach consists in alternating different solutions at the right frequency to fully exploit their potential. In this paper, we first cast the problem as a Markov decision process, where the rewards are a linear function of the recent history of actions, and we show that a policy considering the long-term influence of the recommendations may outperform both fixed-action and contextual greedy policies. We then introduce an extension of the UCRL algorithm (LINUCRL) to effectively balance exploration and exploitation in an unknown environment, and we derive a regret bound that is independent of the number of states. Finally, we empirically validate the model assumptions and the algorithm in a number of realistic scenarios.
The paper changes the model assumptions for recommender systems (RS) to capture phenomena of real world data that has been largely ignored up to now. Instead of assuming a fixed preference over time, the utility of a recommendation is based on the frequency of previous occurrences in a time window w. This captures the fact that humans get bored of repetition. The authors do a great job in showing that this effect occurs in real data. However they still make quite restrictive model assumptions, EDIT{misunderstood this part in the paper, remove comment: i.e. that the utility of an action is only based on the frequency it occurred over the last w times, without taking the positioning into account. The question is whether their model is a good trade-off between computational feasibility and how well it captures the data. To complete the picture, it would be nice to see how much gain in accuracy there is from considering more complex functions that depend on when exactly a recommendation was made. } The main contribution, beside defining a novel model, is an adaptation of UCRL to this problem. The algorithm exploits that the setting is reduced to a deterministic MDP with linear reward function. They use standard techniques to proof an upper bound for this problem, that has favourable dependencies on the problem parameters. The math seems sound, but I haven't checked every detail in the appendix. The paper is well written and easy to understand. The performance of the algorithm is verified in several experiments, which are semi-synthetic. The experimental setup is reasonable and shows convincingly the advantages of the new model.  The biggest downside of this algorithm is, that it only works for small a window w. The authors correctly note that the regret scales fine in w. But they should highlight that the run-time is actually exponential in w. Thereby making this algorithm computationally infeasible even for moderate window sizes.