Locality and Compositionality in Zero-Shot Learning
In this work we study locality and compositionality in the context of learning representations for Zero Shot Learning (ZSL). In order to well-isolate the importance of these properties in learned representations, we impose the additional constraint that, differently from most recent work in ZSL, no pre-training on different datasets (e.g. ImageNet) is performed. The results of our experiment show how locality, in terms of small parts of the input, and compositionality, i.e. how well can the learned representations be expressed as a function of a smaller vocabulary, are both deeply related to generalization and motivate the focus on more local-aware models in future research directions for representation learning.
UPDATE: My recommendation has been borderline because the discussion of the paper about the nature of locality and compositionality seems to be less in-depth than I would have expected, but if the authors will revise the submission to shift the focus of the paper to more focused on analysis and evaluation and weaken the claims that their model exhibits locality and compositionality, I would lean towards an accept as the empirical evaluation is extensive. ---- Summary: This paper aims to investigate the role of locality and compositionality in zero-shot learning. The authors propose a novel evaluation setup that differs from the original zero-shot learning framework in that the model is not allowed to be pretrained on another dataset. The authors propose several methods for learning and visualizing local representations. The overall problem the paper tackles is quite ambitious which involve many parts: defining locality and compositionality, defining constraints for enforcing locality and compositionality, evaluating the existence of locality and compositionality, and evaluating model performance. Because of the extensive nature of the research problem, the coverage of each piece of this research problem in this paper could have been more thorough: for example, I would be really excited for a more in-depth analysis on methods for enforcing locality, but the authors consider only one method, which is the auxiliary loss, and this method does not seem to be applicable for datasets that do not have attribute labels. Given the formalism in section 2.1, I would furthermore be excited about a more in-depth analysis on methods for enforcing compositionality, but the authors only investigate compositionality from the perspective of a weighted average. As a result, the claims about whether enforcing locality and compositionality helps with generalization seem weak, as the authors only do not seem to consider different ways of achieving locality and compositionality -- it is not clear whether their method for enforcing locality (with auxiliary losses) or compositionality (by explicitly performing a weighted average) is representative of enforcing locality and compositionality in general. Furthermore, locality seems to be more of a statement about *ignoring* information (discussed below), which the authors do not explore. As a result, my recommendation is borderline. Perhaps the paper could benefit by reframing it with a more specific focus, with more emphasis on the evaluation, which the paper seems to do well. I would recommend that the authors either (1) perform a more extensive analysis of methods for enforcing locality and compositionality or (2) shift the focus of the paper to more focused on analysis and evaluation and weaken the claims that their model exhibits locality and compositionality, as there is not enough evidence to tell whether their method for enforcing locality and compositionality is the best way of doing so. Strengths: - The paper is well-motivated and tackles an important problem. - The empirical evaluation is extensive. - The framework of not relying on pretrained features is a novel contribution. Weaknesses: - There does not seem to be a comparison between a compositional model and a non-compositional model in section 5.3. Would the authors be able to provide such an analysis? Otherwise it is difficult to tell exactly how and to what extent compositionality helps. - I would have assumed that enforcing locality should actually be a problem of *ignoring* information, rather than *predicting* labeled information. However, the auxiliary loss introduction in section 4.1 only seems to enforce that local features (which by default are local from the CNN) are predictive of attributes/class, rather than enforcing that these features ignore information that is not local to the particular image patch they are modeling. Would the authors be able to comment on this point, as well as provide an empirical analysis of how *unpredictive* the features of their model are on image patches elsewhere?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Locality and Compositionality in Zero-Shot Learning
In this work we study locality and compositionality in the context of learning representations for Zero Shot Learning (ZSL). In order to well-isolate the importance of these properties in learned representations, we impose the additional constraint that, differently from most recent work in ZSL, no pre-training on different datasets (e.g. ImageNet) is performed. The results of our experiment show how locality, in terms of small parts of the input, and compositionality, i.e. how well can the learned representations be expressed as a function of a smaller vocabulary, are both deeply related to generalization and motivate the focus on more local-aware models in future research directions for representation learning.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



UPDATE: My recommendation has been borderline because the discussion of the paper about the nature of locality and compositionality seems to be less in-depth than I would have expected, but if the authors will revise the submission to shift the focus of the paper to more focused on analysis and evaluation and weaken the claims that their model exhibits locality and compositionality, I would lean towards an accept as the empirical evaluation is extensive. ---- Summary: This paper aims to investigate the role of locality and compositionality in zero-shot learning. The authors propose a novel evaluation setup that differs from the original zero-shot learning framework in that the model is not allowed to be pretrained on another dataset. The authors propose several methods for learning and visualizing local representations. The overall problem the paper tackles is quite ambitious which involve many parts: defining locality and compositionality, defining constraints for enforcing locality and compositionality, evaluating the existence of locality and compositionality, and evaluating model performance. Because of the extensive nature of the research problem, the coverage of each piece of this research problem in this paper could have been more thorough: for example, I would be really excited for a more in-depth analysis on methods for enforcing locality, but the authors consider only one method, which is the auxiliary loss, and this method does not seem to be applicable for datasets that do not have attribute labels. Given the formalism in section 2.1, I would furthermore be excited about a more in-depth analysis on methods for enforcing compositionality, but the authors only investigate compositionality from the perspective of a weighted average. As a result, the claims about whether enforcing locality and compositionality helps with generalization seem weak, as the authors only do not seem to consider different ways of achieving locality and compositionality -- it is not clear whether their method for enforcing locality (with auxiliary losses) or compositionality (by explicitly performing a weighted average) is representative of enforcing locality and compositionality in general. Furthermore, locality seems to be more of a statement about *ignoring* information (discussed below), which the authors do not explore. As a result, my recommendation is borderline. Perhaps the paper could benefit by reframing it with a more specific focus, with more emphasis on the evaluation, which the paper seems to do well. I would recommend that the authors either (1) perform a more extensive analysis of methods for enforcing locality and compositionality or (2) shift the focus of the paper to more focused on analysis and evaluation and weaken the claims that their model exhibits locality and compositionality, as there is not enough evidence to tell whether their method for enforcing locality and compositionality is the best way of doing so. Strengths: - The paper is well-motivated and tackles an important problem. - The empirical evaluation is extensive. - The framework of not relying on pretrained features is a novel contribution. Weaknesses: - There does not seem to be a comparison between a compositional model and a non-compositional model in section 5.3. Would the authors be able to provide such an analysis? Otherwise it is difficult to tell exactly how and to what extent compositionality helps. - I would have assumed that enforcing locality should actually be a problem of *ignoring* information, rather than *predicting* labeled information. However, the auxiliary loss introduction in section 4.1 only seems to enforce that local features (which by default are local from the CNN) are predictive of attributes/class, rather than enforcing that these features ignore information that is not local to the particular image patch they are modeling. Would the authors be able to comment on this point, as well as provide an empirical analysis of how *unpredictive* the features of their model are on image patches elsewhere?
assistant
UPDATE: My recommendation has been borderline because the discussion of the paper about the nature of locality and compositionality seems to be less in-depth than I would have expected, but if the authors will revise the submission to shift the focus of the paper to more focused on analysis and evaluation and weaken the claims that their model exhibits locality and compositionality, I would lean towards an accept as the empirical evaluation is extensive. ---- Summary: This paper aims to investigate the role of locality and compositionality in zero-shot learning. The authors propose a novel evaluation setup that differs from the original zero-shot learning framework in that the model is not allowed to be pretrained on another dataset. The authors propose several methods for learning and visualizing local representations. The overall problem the paper tackles is quite ambitious which involve many parts: defining locality and compositionality, defining constraints for enforcing locality and compositionality, evaluating the existence of locality and compositionality, and evaluating model performance. Because of the extensive nature of the research problem, the coverage of each piece of this research problem in this paper could have been more thorough: for example, I would be really excited for a more in-depth analysis on methods for enforcing locality, but the authors consider only one method, which is the auxiliary loss, and this method does not seem to be applicable for datasets that do not have attribute labels. Given the formalism in section 2.1, I would furthermore be excited about a more in-depth analysis on methods for enforcing compositionality, but the authors only investigate compositionality from the perspective of a weighted average. As a result, the claims about whether enforcing locality and compositionality helps with generalization seem weak, as the authors only do not seem to consider different ways of achieving locality and compositionality -- it is not clear whether their method for enforcing locality (with auxiliary losses) or compositionality (by explicitly performing a weighted average) is representative of enforcing locality and compositionality in general. Furthermore, locality seems to be more of a statement about *ignoring* information (discussed below), which the authors do not explore. As a result, my recommendation is borderline. Perhaps the paper could benefit by reframing it with a more specific focus, with more emphasis on the evaluation, which the paper seems to do well. I would recommend that the authors either (1) perform a more extensive analysis of methods for enforcing locality and compositionality or (2) shift the focus of the paper to more focused on analysis and evaluation and weaken the claims that their model exhibits locality and compositionality, as there is not enough evidence to tell whether their method for enforcing locality and compositionality is the best way of doing so. Strengths: - The paper is well-motivated and tackles an important problem. - The empirical evaluation is extensive. - The framework of not relying on pretrained features is a novel contribution. Weaknesses: - There does not seem to be a comparison between a compositional model and a non-compositional model in section 5.3. Would the authors be able to provide such an analysis? Otherwise it is difficult to tell exactly how and to what extent compositionality helps. - I would have assumed that enforcing locality should actually be a problem of *ignoring* information, rather than *predicting* labeled information. However, the auxiliary loss introduction in section 4.1 only seems to enforce that local features (which by default are local from the CNN) are predictive of attributes/class, rather than enforcing that these features ignore information that is not local to the particular image patch they are modeling. Would the authors be able to comment on this point, as well as provide an empirical analysis of how *unpredictive* the features of their model are on image patches elsewhere?