One important aspect of generalization in machine learning involves reasoning about previously seen data in new settings. Such reasoning requires learning disentangled representations of data which are interpretable in isolation, but can also be combined in a new, unseen scenario. To this end, we introduce the contextaware learner, a model based on the variational autoencoding framework, which can learn such representations across data sets exhibiting a number of distinct contexts. Moreover, it is successfully able to combine these representations to generate data not seen at training time. The model enjoys an exponential increase in representational ability for a linear increase in context count. We demonstrate that the theory readily extends to a meta-learning setting such as this, and describe a fully unsupervised model in complete generality. Finally, we validate our approach using an adaptation with weak supervision.
The authors propose an extension to the Neural Statistician which can model contexts with multiple partially overlapping features. This model can explain datasets by taking into account covariate structure needed to explain away factors of variation and it can also share this structure partially between datasets.
A particularly interesting aspect of this model is the fact that it can learn these context c as features conditioned on meta-context a, which leads to a disentangled representation.
This is also not dissimilar to ideas used in 'Bayesian Representation Learning With Oracle Constraints' Karaletsos et al 2016 where similar contextual features c are learned to disentangle representations over observations and implicit supervision.
The authors provide a clean variational inference algorithm to learn their model. However, a key problem is the following: the nature of the discrete variables being used makes them hard to be inferred with variational inference. The authors mention categorical reparametrization as their trick of choice, but do not go into empirical details int heir experiments regarding the success of this approach. In fact, it would be interesting to study which level of these variables could be analytically collapsed (such as done in the Semi-Supervised learning work by Kingma et al 2014) and which ones can be sampled effectively using a form of reparametrization.
This also touches on the main criticism of the paper: While the model technically makes sense and is cleanly described and derived,  the empirical evaluation is on the weak side and the rich properties of the model are not really shown off. It would be interesting if the authors could consider adding a more illustrative experiment and some more empirical results regarding inference in this model and the marginal structures that can be learned with this model in controlled toy settings.
Can the model recover richer structure that was imposed during data generation? How limiting is the learning of a?
How does the likelihood of the model behave under the circumstances?
The experiments do not really convey how well this all will work in practice.