Towards Effective and Efficient Zero-shot Learning by Fine-tuning with Task Descriptions
While current machine learning models have achieved great success with labeled data, we have to deal with classes that have little or no training data in many real-world applications. This leads to the study of zero-shot learning. The typical approach in zero-shot learning is to embed seen and unseen classes into a shared space using class meta-data, and construct classifiers on top of that. Yet previous methods either still require significant manual labor in obtaining useful meta-data, or utilize automatically collected meta-data while trading in performance. To achieve satisfactory performance under practical meta-data efficiency constraint, we propose 	extbf{N	extsuperscript{3}} (	extbf{N}eural 	extbf{N}etworks from 	extbf{N}atural Language), a meta-model that maps natural language class descriptions to corresponding neural network classifiers. N	extsuperscript{3} leverages readily available online documents combined with pretrained language representations such as BERT to obtain expressive class embeddings. In addition, N	extsuperscript{3} generates parameter adaptations for pretrained neural networks using these class embeddings, effectively ``finetuneing** the network to classify unseen classes. Our experiments show that N	extsuperscript{3} is able to outperform previous methods across 8 different benchmark evaluations and we show through ablation studies the contribution of each model component. To offer insight into how N	extsuperscript{3} ``finetunes** the pretrained network, we also performed a range of qualitative and quantitative analysis. Our code will be released after the review period.
Summary: This paper proposes a meta-model called Neural Networks from Natural language (N^3) for zero-shot learning. This model maps a set of class object descriptions to neural network parameters to create zero-shot classifiers, which contains two modules: (1) SEM encodes class descriptions into corresponding embeddings; (2) PAM fine-tunes the generated parameters in the SEM. The authors conduct experiments on 4 popular benchmark datasets and achieve best results compared with baselines. Strengths: (1) This paper proposes a good Parameter Adaptation Module (PAM) to generate parameter adaptations to fine-tune all layers in the pretrained network based on the class embeddings, to classify unseen classes. (2) This paper achieves good results compared with baselines. Weaknesses: (1) I am not sure the effectiveness of the N^3 according to the results in Table 3, 4, because the BERT module may play a major role. (2) Although PDCNN (2015) and GAZSL (2017) are very strong baselines, this paper does not compare the work published in 2018 or 2019. (3) There are many grammatical mistakes in this paper. I have some following major concerns about the paper: (1) This paper conducts extensive experiments to demonstrate the performance of N^3 in Table 2, but why not compare N^3 with the work published in 2018 or 2019? (2) In Table 3, this paper studies what extent does the superior performance of N^3 depend on gamma and mu, but why not show the results with gamma =1, mu=0 to demonstrate the effectiveness of PAM or fine-tuning? (3) In Table 4, I can come to a conclusion that BERT plays a great role in your model. To better demonstrate the effectiveness of N^3, I think you should show the accuracy of the baselines and N^3 with ELMo and Glove embeddings separately in Table 2. (4) This paper does not explain the results clearly such as Figure 5 (a) and (b). (5) There are many grammatical mistakes in this paper, such as the sentence “we also perform ...... and demonstrated ......” in Section 1.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Towards Effective and Efficient Zero-shot Learning by Fine-tuning with Task Descriptions
While current machine learning models have achieved great success with labeled data, we have to deal with classes that have little or no training data in many real-world applications. This leads to the study of zero-shot learning. The typical approach in zero-shot learning is to embed seen and unseen classes into a shared space using class meta-data, and construct classifiers on top of that. Yet previous methods either still require significant manual labor in obtaining useful meta-data, or utilize automatically collected meta-data while trading in performance. To achieve satisfactory performance under practical meta-data efficiency constraint, we propose 	extbf{N	extsuperscript{3}} (	extbf{N}eural 	extbf{N}etworks from 	extbf{N}atural Language), a meta-model that maps natural language class descriptions to corresponding neural network classifiers. N	extsuperscript{3} leverages readily available online documents combined with pretrained language representations such as BERT to obtain expressive class embeddings. In addition, N	extsuperscript{3} generates parameter adaptations for pretrained neural networks using these class embeddings, effectively ``finetuneing** the network to classify unseen classes. Our experiments show that N	extsuperscript{3} is able to outperform previous methods across 8 different benchmark evaluations and we show through ablation studies the contribution of each model component. To offer insight into how N	extsuperscript{3} ``finetunes** the pretrained network, we also performed a range of qualitative and quantitative analysis. Our code will be released after the review period.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



Summary: This paper proposes a meta-model called Neural Networks from Natural language (N^3) for zero-shot learning. This model maps a set of class object descriptions to neural network parameters to create zero-shot classifiers, which contains two modules: (1) SEM encodes class descriptions into corresponding embeddings; (2) PAM fine-tunes the generated parameters in the SEM. The authors conduct experiments on 4 popular benchmark datasets and achieve best results compared with baselines. Strengths: (1) This paper proposes a good Parameter Adaptation Module (PAM) to generate parameter adaptations to fine-tune all layers in the pretrained network based on the class embeddings, to classify unseen classes. (2) This paper achieves good results compared with baselines. Weaknesses: (1) I am not sure the effectiveness of the N^3 according to the results in Table 3, 4, because the BERT module may play a major role. (2) Although PDCNN (2015) and GAZSL (2017) are very strong baselines, this paper does not compare the work published in 2018 or 2019. (3) There are many grammatical mistakes in this paper. I have some following major concerns about the paper: (1) This paper conducts extensive experiments to demonstrate the performance of N^3 in Table 2, but why not compare N^3 with the work published in 2018 or 2019? (2) In Table 3, this paper studies what extent does the superior performance of N^3 depend on gamma and mu, but why not show the results with gamma =1, mu=0 to demonstrate the effectiveness of PAM or fine-tuning? (3) In Table 4, I can come to a conclusion that BERT plays a great role in your model. To better demonstrate the effectiveness of N^3, I think you should show the accuracy of the baselines and N^3 with ELMo and Glove embeddings separately in Table 2. (4) This paper does not explain the results clearly such as Figure 5 (a) and (b). (5) There are many grammatical mistakes in this paper, such as the sentence “we also perform ...... and demonstrated ......” in Section 1.
assistant
Summary: This paper proposes a meta-model called Neural Networks from Natural language (N^3) for zero-shot learning. This model maps a set of class object descriptions to neural network parameters to create zero-shot classifiers, which contains two modules: (1) SEM encodes class descriptions into corresponding embeddings; (2) PAM fine-tunes the generated parameters in the SEM. The authors conduct experiments on 4 popular benchmark datasets and achieve best results compared with baselines. Strengths: (1) This paper proposes a good Parameter Adaptation Module (PAM) to generate parameter adaptations to fine-tune all layers in the pretrained network based on the class embeddings, to classify unseen classes. (2) This paper achieves good results compared with baselines. Weaknesses: (1) I am not sure the effectiveness of the N^3 according to the results in Table 3, 4, because the BERT module may play a major role. (2) Although PDCNN (2015) and GAZSL (2017) are very strong baselines, this paper does not compare the work published in 2018 or 2019. (3) There are many grammatical mistakes in this paper. I have some following major concerns about the paper: (1) This paper conducts extensive experiments to demonstrate the performance of N^3 in Table 2, but why not compare N^3 with the work published in 2018 or 2019? (2) In Table 3, this paper studies what extent does the superior performance of N^3 depend on gamma and mu, but why not show the results with gamma =1, mu=0 to demonstrate the effectiveness of PAM or fine-tuning? (3) In Table 4, I can come to a conclusion that BERT plays a great role in your model. To better demonstrate the effectiveness of N^3, I think you should show the accuracy of the baselines and N^3 with ELMo and Glove embeddings separately in Table 2. (4) This paper does not explain the results clearly such as Figure 5 (a) and (b). (5) There are many grammatical mistakes in this paper, such as the sentence “we also perform ...... and demonstrated ......” in Section 1.