Current deep learning based text classification methods are limited by their ability to achieve fast learning and generalization when the data is scarce. We address this problem by integrating a meta-learning procedure that uses the knowledge learned across many tasks as an inductive bias towards better natural language understanding. Inspired by the Model-Agnostic Meta-Learning framework (MAML), we introduce the Attentive Task-Agnostic Meta-Learning (ATAML) algorithm for text classification. The proposed ATAML is designed to encourage task-agnostic representation learning by way of task-agnostic parameterization and facilitate task-specific adaptation via attention mechanisms. We provide evidence to show that the attention mechanism in ATAML has a synergistic effect on learning performance. Our experimental results reveal that, for few-shot text classification tasks, gradient-based meta-learning approaches outperform popular transfer learning methods. In comparisons with models trained from random initialization, pretrained models and meta trained MAML, our proposed ATAML method generalizes better on single-label and multi-label classification tasks in miniRCV1 and miniReuters-21578 datasets.
Summary of paper: For the few shot text classification task, train a model with MAML where only a subset of parameters (attention parameters in this case) are updated in the inner loop of MAML. The empirical results suggest that this improves over the MAML baseline.
I found this paper confusingly written. The authors hop between a focus on meta-learning to a focus on attention, and it remains unclear to me how these are connected. The description of models is poor -- for example, the ablation mentioned in 4.5.3 is still confusing to me (if the attention parameters are not updated in the inner loop of MAML, then what is?). Furthermore, even basic choices of notation, like A with a bar underneath in a crowded table, seem poorly thought out.
I find the focus on attention a bit bizarre. It's unclear to me how any experiments in the paper suggest that attention is a critical aspect of meta-learning in this model. The TAML baseline (without attention) underperforms the ATAML model (with attention), but all that means is that attention improves representational power, which is not surprising. Why is attention considered an important aspect of meta learning?
To me, the most interesting aspect of this work is the idea of not updating every parameter in the MAML inner loop. So far, I've seen all MAML works update all parameters. The experiments suggest that updating a small subset of parameters can improve results significantly in the 1-shot regime, but the gap between normal MAML and the subset MAML is much smaller in the 5-shot regime. This result suggests updating a subset of parameters can serve as a method to combat overfitting, as the 1-shot regime is much more data constrained than the 5-shot regime.
It's unfortunate that the authors do not dig further down this line of reasoning. When does the gap between MAML on all parameters and only on a subset of parameters become near-zero? Does the choice of the subset of parameters matter? For example, instead of updating the attention weights, what happens if the bottommost weights are updated? How would using pretrained parameters (e.g., language modeling pretraining) in meta-learning affect these results? In general, what can be learned about overfitting in MAML?
To conclude, the paper is not written well and has a distracting focus on attention. While it raises an interesting question about MAML and overfitting, it does not have the experiments needed to explore this topic well.