In this work we study locality and compositionality in the context of learning representations for Zero Shot Learning (ZSL). In order to well-isolate the importance of these properties in learned representations, we impose the additional constraint that, differently from most recent work in ZSL, no pre-training on different datasets (e.g. ImageNet) is performed. The results of our experiments show how locality, in terms of small parts of the input, and compositionality, i.e. how well can the learned representations be expressed as a function of a smaller vocabulary, are both deeply related to generalization and motivate the focus on more local-aware models in future research directions for representation learning.
###  Summary 
- The paper tries to understand what learning principles can lead to representations that allow zero-shot learning/generalization. 
- It explores the impact of two properties -- locality and compositionality -- on ZSL performance. 
- To encourage locality (and also compositionality since they can overlap), the paper uses an auxiliary loss at an earlier layer (when the receptive field of features is small) to predict local attributes. 
- For interpreting local features, the paper computes MI between the global feature map of one image and local features from a different image of the same class and visualizes it using a heatmap.  
- It uses normalized TRE to measure compositionality (Normalized to take into account that some methods are biased towards learning more compositional representations.)
- It compares supervised, unsupervised, and self-supervised representation learning methods and studies the impact of locality and compositionality on ZSL performance for these methods. 
### Decision and reasons
I vote for a weak accept. However, I think it's a good paper and does a lot of things right. I'm willing to increase my score if the authors can convincingly answer my questions.  
Positives: 
1- The proposed zero-shot learning from scratch setting is a step in the right direction for focusing on uncovering general learning principles. 
2- Locality and compositionality are sensible goals for good representations. The paper defines both and explores their importance with clear and novel experiment-designs and metrics. The experiments are also well conducted.
Negatives: 
1- The paper makes a lot of observations, but sometimes does not even try to explain some unexpected observations. 
2- CMDIM is presented as a self-supervised algorithm but seems to require class labels. 
### Supporting arguments for the reasons for the decision.
Positives: 
1- I agree with the paper that ZSL setting is more about uncovering learning principles and less about constructing practical systems that can do well in zero-shot settings. In a practical setting, it doesn't make much sense to zero-shot learning anyway. I also agree that current ZSL work is focusing too much on pushing the state-of-the-art using pre-trained imagenet features and missing the actual goal of the problem setting (I wouldn't say we shouldn't do that at all, however. Some learning principles may require a significant amount of data to learn good representations for zero-shot learning, and imagenet pretraining is a good proxy for that). This paper acts as a good reminder that ZSL research should be done keeping in mind the goal of ZSL.
2- The paper first defines locality and compositionality. It then uses interesting and meaningful metrics for empirically testing if these properties correlate with zero-shot performance. I found the experiment designs to be clever (such as computing parts F-1 score using boolean maps, visualizing MI between local and global features of images from the same class, etc).  I think this paper will act as an important reference for motivating future work on learning more modular representations. 
Negatives: 
1- The paper doesn't try to explain the possible reasons behind some observations. For example, why does locality not correlate with ZSL performance of generative models? Why does LC loss hurt performance for CMDIM? 
2- CMDIM draws positive samples from other images of the same class. As a result, it's not a truly unsupervised learning method. The direct comparison of CMDIM to DIM seems unfair. 
### Questions
Q1-  Computation of the local classification and attribute auxiliary loss is a bit unclear. How is it assured that a certain attribute is present in the local feature when computing the auxiliary attribute-loss? 
Q2- Did the authors try a baseline in which AC is also trained on the global representation in the context of Figure 2? The extra information about the parts may be the reason behind better ZSL performance, and a global AC could improve ZSL performance without improving Parts F1 score (Although a more likely outcome is that a global AC would increase both the parts F1 score and the ZSL accuracy.)
Q3- For VAEs, the local F-1 score does not correlate with ZSL performance. This seems to contradict the idea locality leads to better ZSL performance. Is there an explanation for this? The paper just glances over this by calling this 'interesting.' 
Q4- Can CMDIM be considered a self-supervised algorithm? Doesn't it need class labels to draw positive samples from the same class? 
### Other minor remarks 
- "Formally, f(x) \elem R is compositional if it can be expressed as a combination of the elements of ..."
The definition would be more clear if the authors could mention some reasonable combination operators here (such as weighted average etc).
- "However, this choice in architecture does not guarantee locality, as CNN representations could only contain “global” information, such as the class or color of the object, despite having a limited receptive field. "
I'm not sure what this means. Why are CNN representations restricted to 'only' global information?
### Update after author's response
The author's response fixes some minor clarity issues. I think this work is a good contribution and should be accepted. I've updated my score accordingly.