We study neural-linear bandits for solving problems where both exploration and representation learning play an important role. Neural-linear bandits leverage the representation power of deep neural networks and combine it with efficient exploration mechanisms, designed for linear contextual bandits, on top of the last hidden layer. Since the representation is being optimized during learning, information regarding exploration with "old" features is lost. Here, we propose the first limited memory neural-linear bandit that is resilient to this catastrophic forgetting phenomenon. We perform simulations on a variety of real-world problems, including regression, classification, and sentiment analysis, and observe that our algorithm achieves superior performance and shows resilience to catastrophic forgetting.
This paper proposes a neural linear bandits algorithm that is resilient to catastrophic forgetting when using limited memory.
The proposed algorithm Alg. 2 is similar to Thompson sampling for linear contextual bandits, Alg. 1, but using the last layer activation vectors as a linear feature, and also a different way of updating noise parameter prior and posterior is used based on Bayesian linear regression Eq. (2).
Alg. 2 also works with limited memory of history data, therefore after every time, the memory is refreshed, likelihood matching is used to calculate new Phi to make the likelihood (mean and variance) of reward estimation the same as it for the old feature. For mean matching, minimizing MSE Eq. (3) is used and for variance, solving PSD problem Eq. (6) is used.
The complexity of this algorithm is analyzed. And experiments are conducted to show that the proposed method is resilient to catastrophic forgetting and can achieve good cumulative reward results.
The proposed method is reasonable and the results look promising. However, I found several weak points as follows.
1. As mentioned Bayesian linear regression Eq. (2) is used to update noise prior and posterior, but this update has no theoretical guarantees as mentioned. As an algorithm mainly works under bandit settings, this is kind of undesirable.
2. This algorithm works with neural network-based features, but it is in nature not scalable as shown in the complexity analysis (linear dependence on action number). The linear feature is just replaced by the last layer activation of NNs. From this perspective, the experimental results just justify again that the NN feature is somehow powerful, which is as expected.
3. The likelihood matching can deal with catastrophic forgetting with limited history memory, which looks good. But the fact that it actually works for linear feature (last layer activation) together with realization assumption weakens this contribution a lot. The authors find using Eq. (3) is better than the exact mean matching Eq. (5), and there is no explanation for this, which kind of shows the proposed likelihood matching probably is not a good way when using full NNs rather than just linear features (last layer). On the other hand, the SDP seems also can only work under linear feature settings, and is not promising to be generalized to fully update for NNs.
Overall, this is a reasonable paper. However, on the one hand, as an algorithm mainly works under bandit settings, it is a lack of theoretical support. On the other hand, the linear feature setting weakens the contribution of likelihood matching to deal with catastrophic forgetting with limited memory. There are some questions of the proposed mean matching, and the matching is not able to generalize.