Interactive Submodular Bandit
In many machine learning applications, submodular functions have been used as a model for evaluating the utility or payoff of a set such as news items to recommend, sensors to deploy in a terrain, nodes to influence in a social network, to name a few. At the heart of all these applications is the assumption that the underlying utility/payoff function is known a priori, hence maximizing it is in principle possible. In many real life situations, however, the utility function is not fully known in advance and can only be estimated via interactions. For instance, whether a user likes a movie or not can be reliably evaluated only after it was shown to her. Or, the range of influence of a user in a social network can be estimated only after she is selected to advertise the product. We model such problems as an interactive submodular bandit optimization, where in each round we receive a context (e.g., previously selected movies) and have to choose an action (e.g., propose a new movie). We then receive a noisy feedback about the utility of the action (e.g., ratings) which we model as a submodular function over the context-action space. We develop SM-UCB that efficiently trades off exploration (collecting more data) and exploration (proposing a good action given gathered data) and achieves a O( √ T ) regret bound after T rounds of interaction. More specifically, given a bounded-RKHS norm kernel over the context-action-payoff space that governs the smoothness of the utility function, SM-UCB keeps an upperconfidence bound on the payoff function that allows it to asymptotically achieve no-regret. Finally, we evaluate our results on four concrete applications, including movie recommendation (on the MovieLense data set), news recommendation (on Yahoo! Webscope dataset), interactive influence maximization (on a subset of the Facebook network), and personalized data summarization (on Reuters Corpus). In all these applications, we observe that SM-UCB consistently outperforms the prior art.
Pros:
        - The paper combines submodular function maximization with contextual bandit, and propose the problem of interactive submodular bandit formulation and SM-UCB solution. It includes several existing studies as special cases.
        - The empirical evaluation considers several application contexts and demonstrates the effectiveness of the SM-UCB algorithm.
        Cons:
        - The discussion on the key RKHS condition is not enough, and its applicability for actual applications is unclear. More importantly, there is no discussion about constant B for the RKHS norm for various applications, and also the maximum information gain \gamma_t, but both of them are needed in the algorithm.
        - The technical analysis is heavily dependent on prior study, in particular [33]. The additional technical contribution is small.

        The paper studies a contextual bandit learning setting, where in each round a context \phi is presented, which corresponds to an unknown submodular set function f_\phi. The user needs to select one item x to add to the existing selected set S_\phi, with the purpose of maximizing the marginal gain of x given S_\phi. After selecting x, the user obtains a noisy feedback of the marginal gain. The regret is comparing against the (1-1/e) fraction of the optimal offline solution, when all the submodular functions for all contexts are known. The key assumption to make the learning work is that the marginal contribution \Delta in terms of the subset S, the new item x, and the context \phi, has a bounded RKHS norm. Intuitively, this norm makes sure that similar contexts, similar subsets, and similar item would give similar marginal results. 
        The authors propose the SM-UCB algorithm to solve the problem. The proposed algorithm and its regret analysis is heavily based on the prior work on contextual Gaussian process bandit optimization [33]. In fact, it looks like all the authors need to do is to treat existing subset S already selected as part of the context, and then plugging it into the algorithm and the analysis of [33]. The proof of Theorem 1 given in the supplementary material is only to connect their current regret definition with the regret definition of [33], using the submodularity property. Then they just apply Theorem 1 of [33] to obtain the result. Therefore, from the technical point of view, the paper's new contribution is small.
        [Note: After reading the authors' feedback, I get a bit clearer understanding, in that the difference in model from [33] seems to be that the current action may affect future contexts. However, I am still not quite sure how much the analysis differs from [33], since there is only a two-and-a-half page analysis in the supplementary material, and it mostly relies on [33] to achieve the main result. Please clarify the technical contribution of the current paper, and what are the novelty in the analysis that differs from [33]]
        Thus, the main contribution of the paper, in my view, is to connecting submodular maximization with prior work on contextual bandit and give the formulation and the result of interactive submodular bandit problem. Their empirical evaluation covers several application cases and demonstrate advantages of the AM-UCB algorithm, giving readers some concrete examples on how to apply their general results in different contexts. 
        However, one important thing the authors did not discuss much is the key assumption on the bounded RKHS norm. First, RKHS norm should be better defined and explained. Second, its implication on several application contexts should be discussed. Third, the choice of kernel functions should be discussed. More importantly, the bound B should be evaluated for the applications they discuss and use in the experiment section. This is because this complexity measure B is not only used in the regret bound, but also appears as part of \beta_t in the algorithm. Similar situation occurs for the maximum information gain paramter \gamma_t. The authors do not discuss at all how to obtain these parameters for the algorithm. Without these parameters, it is not even clear how the authors evaluate their SM-UCB algorithm in the experiments. This is a serious ommission, and the readers would not be able to apply their algorithm, or reproduce their experiments without knowing how to properly set B and \gamma_t. 
        Furthermore, In the empirical evaluation section, the authors use linear kernel and Jaccard kernel, but why these are reasonable ones? What are their implications to the application context? What are the other possible kernels to use? In general, I do not have a good idea how to pick a good kernel function for an application, and what are the impact and limitations of this assumption. I think this should be the key contribution of the paper, since the other technical contribution is minor, but the authors fail to provide satisfactory answers in this regard.
        Other minor comments:
        - line 133, "T_i" --> "T_j"
        - line 164, I(y_A | \Delta) --> I(y_A; \Delta)
        - Related work
        It would be nice if the authors could compare their approach with other related approaches combining multi-armed bandit with combinatorial optimization tasks, such as
        a) combinatorial multi-armed bandit with application to influence maximization and others: Combinatorial Multi-Armed Bandit and Its Extension to Probabilistically Triggered Arms Wei Chen, Yajun Wang, Yang Yuan, Qinshi Wang, in Journal of Machine Learning Research, 2016
        b) Online influence maximization. Siyu Lei, Silviu Maniu, Luyi Mo, Reynold Cheng, and Pierre Senellart. In KDD, 2015.
        c) Stochastic Online Greedy Learning with Semi-bandit Feedbacks. Tian Lin, Jian Li, Wei Chen, In NIPS'2015