In this paper, we solve the arms exponential exploding issues in multivariate Multi-Armed Bandit (Multivariate-MAB) problem when the arm dimension hierarchy is considered. We propose a framework called path planning (TS-PP) which utilizes decision graph/trees to model arm reward success rate with m-way dimension interaction, and adopts Thompson sampling (TS) for heuristic search of arm selection. Naturally, it is quite straightforward to combat the curse of dimensionality using a serial processes that operates sequentially by focusing on one dimension per each process. For our best acknowledge, we are the first to solve Multivariate-MAB problem using graph path planning strategy and deploying alike Monte-Carlo tree search ideas. Our proposed method utilizing tree models has advantages comparing with traditional models such as general linear regression. Simulation studies validate our claim by achieving faster convergence speed, better efficient optimal arm allocation and lower cumulative regret.
In this paper, the authors address the curse of dimensionality in Multivariate Multi-Armed Bandits by framing the problem as a sequential planning problem.
Their contributions are the following:
1. They propose to solve the Multivariate MAB sequentially across each dimension, which I think is a reasonable and not very common approach in the literature compared to e.g. structured bandits, even though the authors themselves call this approach "natural" and "quite straightforward".
2. They propose a framework based on a Monte-Carlo Tree Search procedure to solve this sequential decision problem.
3. They introduce several approximations and heuristics to avoid traversing the entire tree and trade-off performance for computational complexity, which they evaluate empirically.
The obtained results seem to support their original claim: the tree-based approaches (e.g. FPF) perform better than the considered baselines. However, this claim is at the same time contradicted by the fact that a depth-one planning procedure with heuristic leaf evaluation (PPF2) reaches similar performances, which indicates that the underlying problem can be solved greedily and does not involve long-term (m-way) interations and limits the interest of the main idea in this paper. This is probably due to the design of experiments which only model pairwise interactions, but then this questions the relevance of the proposed empirical evaluation with respect to the claim.
Generally, though several good ideas were presented, I felt that they were not properly motivated and that the paper generally lacks rigor and clarity. The many variants and modeling decisions proposed seem quite arbitrary, and in the end fail to provide any valuable insight. 
Several aspects of the proposed approach and algorithms were not clear or justified. I will list some of them, grouped by the corresponding contribution:
1. Multivariate MAB framed as a planning problem.
No intuition is provided regarding why it is expected to perform any better than classical N^D-MAB. In this absence of assumption regarding additional structure to the problem considered (such as, say, the independence assumption in D-MABS), is there any reason to think so? (for instance, all the N^D arms are present at the leaves of the tree) The authors only mention in their conclusion that they leverage a "hierarchy dimension structure", but this claimed structure was is never defined nor formalized in the paper. I can only guess that such a structure would involve a particular ordering of dimensions such that shallow ones have more influence on the payoff than the deeper ones? But then, it would be unknown in practice, and the choice of the particular ordering used in TS-PP is expected to have a crucial influence on its performance. Yet, the sensitivity of their algorithms to the ordering is never discussed by the authors -they choose it arbitrarily-  nor evaluated in their experiments.
2. The proposed TS-PP framework
- The authors propose using MCTS with TS instead of UCB as a sampling rule, which they claim is quite novel: "Applying TS as node selection policy is not well investigated from our best knowledge". However, the authors do not mention important reference (Bai et al, 2014) until the very end of the paper in their conclusion: "We notice some related work dealing with continuous rewards in this area". Yet, Bai et al. also consider Bernoulli rewards in their experiments and use conjugate Beta priors, exactly like the authors of the present work do. Moreover, the choice of TS rather than UCB is not properly motivated, and the UCT algorithm is not considered as a baseline in the experiments. Hence, there is no basis to assess whether TS is particularly relevant or not for this problem.
- They also propose to repeat S times the candidate searching procedure in Algorithm 1 in order not to be "[stuck] in sub-optimal arms". This property is normally ensured by optimistic sampling rules, it is not clear why it is required here. More importantly, the sensitivity of the Algorithm 1 to S is not evaluated.
- I did not fully understand the Candidates Construction Stage in Algorithm 1. For instance, in the case of FPF, for each search c=1..S (l3) there is a call (l4) to Full Path Finding() to generate a candidate A^c. From what I understand, a random ordering is chosen (Algorithm 2, l2), and a single rollout is performed (l2, the dimensions 1->D are iterated from root to leaf) to return a candidate. When we move to the next search c -> c+1, since a new ordering d_1:D is chosen then the previous tree storing the models for p(fd_i | fd_1...fd_i-1) is not relevant anymore and must be discarded. Does that mean that the priors at the nodes are recomputed for each search/ordering/tree given the entire history, instead of being maintained and updated? The authors do state that "FFP utilizes hidden states from the same decision tree", but I do not see how this is feasible when the dimensions ordering changes at each iteration. 
- This confusion is related to the fact that the update of the Beta posterior does not appear anywhere in Algorithm 1 and 2. This is unfortunate, especially considering that contrary to TS in a normal MAB setting, the observations for an internal node are not i.i.d. since they depend on the underlying planning procedure, which induces a distributional drift as more sample are collected and probably make the derived posterior invalid. Has this issue been considered by the authors?
3. The variants of FPF
The idea behind Partial-PF-m is common in the tree-search literature: to stop at early depths and use a heuristic to estimate the value at the leaf. Here, the suggestion in the particular context of Multivariate-MAB to use TS with an independence assumption (D-MABs) is interesting and sound. However, It requires the knowledge the degree m of interactions, whereas the idea of MCTS is precisely to avoid using such heuristics for leaf evaluation by replacing them without random rollouts. Other methods such as DS start at a leaf and move locally which rather resembles Hill-climbing more closely than actual Tree-Search, and I do not really see how they fit in the TS-PP framework?
Finally, I found the manuscript difficult to read due to many language mistakes and typos.
Minor comments:
- The citation style is incorrect.
- What is the point of introducing contextual MABs in the problem formulation when the context is not going to play any part whatsoever in the presented work? This only brings unnecessary complexity and impedes clarity.
- In the experiments, H=100 replications are performed, which is appreciated, but unfortunately the corresponding confidence intervals or standard deviations are not shown, only the mean performance.
- I found the section 3.1 unclear. Is B_A^T \mu a dot product? Where do the coefficients of B_A or its length M appear in (1)? How is M chosen, how is B_A obtained?
- Figure 1.a is also unclear, the graph seems to be used to define permutations/orderings of dimensions 1..D. But all paths are not valid (eg. d1 -> d2 -> d1), ad random orderings are used in Algorithm 2, so where is this graph used exactly?
=============
After Rebuttal:
On one hand, some of the points mentioned are now clearer (e.g. 1 & 2.2, 2.3).
On the other hand, most of my concerns remain valid, and some were not addressed.
Typically, the dependency to an unknown (and undefined...) optimal ordering and the underlying optimization procedure (random search) are not properly discussed, whereas they seem central to justify the tree-based approach.
Other remaining concerns are more generally the lack of justification and clarity of the proposed framework and implemented variants, and the inadequacy of the experiments with respect to the claim (using m=2 basically means only the first depth of the tree is useful). Therefore, I will maintain my initial score.