Extracting the hidden structure of the external environment is an essential component of intelligent agents and human learning. The real-world datasets that we are interested in are often characterized by the locality: the change in the structural relationship between the data points depending on location in observation space. The local learning approach extracts semantic representations for these datasets by training the embedding model from scratch for each local neighborhood, respectively. However, this approach is only limited to use with a simple model, since the complex model, including deep neural networks, requires a massive amount of data and extended training time. In this study, we overcome this trade-off based on the insight that the real-world dataset often shares some structural similarity between each neighborhood. We propose to utilize the embedding model for the other local structure as a weak form of supervision. Our proposed model, the Local VAE, generalize the Variational Autoencoder to have the different model parameters for each local subset and train these local parameters by the gradient-based meta-learning. Our experimental results showed that the Local VAE succeeded in learning the semantic representations for the dataset with local structure, including the 3D Shapes Dataset, and generated high-quality images.
This paper proposed a local VAE based on the model-agnostic meta-learning concept. Images generated based local VAE are shown to be better than those generated bu \beta-VAE in general in terms of generation quality and disentanglement/compactness/informativeness.
This paper shows an interesting idea of viewing neighbourhood of a data point as a task to adopt the meta-learning concept to  train VAE which gives superior performance.
However, this paper falls short regarding its overall organisation of the paper and the way that different concepts being articulated which make it hard to follow and read. It is not easy to understand the linkage between a task and a neighbourhood being advocated without re-reading the paper for a few times. It seems to me that very substantial effort is needed for the revision to reach the ICLR standard.
The introduction section should be carefully rewritten to explain clearly the notions of locality and structural relationship. As far as I can see in the subsequent sections, the locality/structural relationship seems to be referring to the neighbourhood either in the input space or the latent space. Also, the way of discussing local learning should be elaborated a bit to make it clear what exact notion of local learning  is being presented.
Specific comments:
Page 2: 
“since the most dataset tends to be governed by the consistent rules of the physical world”
->
“since most dataset tends to be governed by the consistent rules of the physical world”
“Such kind of dataset has a multi-scale structure from a local to a global scale.” - some grammatical issue
Page 5:
Line 4: Equation 3 seems not related to meta-learning