Transfer learning using ImageNet pre-trained models has been the de facto approach in a wide range of computer vision tasks. However, fine-tuning still requires task-specific training data. In this paper, we propose N 3 ( N eural N etworks from N atural Language) a new paradigm of synthesizing task-specific neural networks from language descriptions and a generic pre-trained model. N 3 leverages language descriptions to generate parameter adaptations as well as a new task-specific classification layer for a pre-trained neural network, effectively fine-tuning the network for a new task using only language descriptions as input. To the best of our knowledge, N 3 is the first method to synthesize entire neural networks from natural language. Experimental results show that N 3 can out-perform previous natural-language based zero-shot learning methods across 4 different zero-shot image classification benchmarks. We also demonstrate a simple method to help identify keywords in language descriptions leveraged by N 3 when synthesizing model parameters. 1 1 Introduction A person with generic world knowledge can learn to perform a new task based on verbal instructions. On the other hand, despite recent successes in deep  First two authors contributed equally. learning, it remains challenging to re-purpose pretrained visual classification models to recognize a new set of objects without labeling a new training dataset. A natural question emerges from this observation: can a computer also learn to recognize new objects, simply by reading the descriptions of them in natural language? Concretely, can we create visual classifiers using language descriptions of the objects of interest? In this paper, we introduce a new paradigm for synthesizing task-specific neural networks for image classification simply from language descriptions of the relevant objects. We propose N 3 N eural N etworks from N atural Language, a meta-model that takes a list of object descriptions as input to produce a classification model for these objects, as illustrated in Figure 1. The capability of producing task-specific neural network from language descriptions makes N 3 ideal to a wide range of zero-shot tasks (Wah et al., 2011; Zhu et al., 2017; Elhoseiny et al., 2017; Zhu et al., 2017; Nilsback and Zisserman, 2006) where we do not have visual training data but can still easily obtain language descriptions of the objects of interest. Prior zero-shot learning methods aim to achieve a similar goal of applying pre-trained networks on unseen classes. In zero-shot image classification, a typical method of generalizing to unseen classes is to construct class embeddings to augment the classification layer of the pre-trained network or take retrieval-based approaches while utilizing generic visual features from pre-trained networks (Akata et al., 2015; Kodirov et al., 2017; Elhoseiny et al., 2013; Lei Ba et al., 2015; Reed et al., 2016). Extending on the idea of generating classification layers for the pre-trained network, N 3 modifies the parameters of all layers in the pre-trained network. While the underlying pre-trained network in previous approaches only extracts generic visual features, N 3 makes it possible to extract task-specific ones. This approach effectively increases the capacity at which semantic information in the descriptions can affect the pre-trained network. In our experiments, we evaluated our proposed N 3 method with 4 popular zero-shot image classification benchmark datasets. We performed ablation studies to understand the importance of synthesizing task-specific feature extractors, the necessity of a pre-trained visual classification model and the effects of language representation choices on the efficacy of N 3 . In addition, we provide a simple approach to help interpret what aspects in the language descriptions are N 3 -generated models examining when making predictions. 1. We propose a novel meta-model N 3 to synthesize task-specific neural network models using natural language descriptions. 2. We demonstrate N 3 's superior efficacy in solving natural language guided zero-shot learning problems. Our analysis shows that N 3 's ability to tailor neural network models to extract task-specific features plays an important role in achieving such superior accuracy. 3. We show that N 3 can aid the interpretation of predictions of synthesized models as they can be traced back to both supportive and refutative evidence within language descriptions. In this section, we are situating our work in the context of zero-shot learning and dynamic parameter generation for neural networks. Zero-shot Learning Zero-shot learning studies how we can generalize our models to perform well for tasks without any labeled training data at all. Achieving classification accuracy above chance-level in such scenarios requires modeling the relationships between the seen classes during training and the unseen classes during testing. A typical and effective method is to manually engineer class attribute vectors to obtain representations of seen and unseen classes in a shared attribute space (Duan et al., 2012; Kankuekul et al., 2012; Parikh and Grauman, 2011; Zhang and Saligrama, 2016; Akata et al., 2016). Yet such a method requires laborious engineering of class attributes, which is not feasible for large-scale and/or fine-grained classification tasks (Russakovsky et al., 2015; Khosla et al., 2011; Welinder et al., 2010). Hence, there is also work in zero-shot learning that attempts to leverage textual data as object class representations (Lei Ba et al., 2015; Elhoseiny et al., 2013). The majority of these models are committed to embedding-based retrieval approaches, where classification is re-formulated as retrieving the class embedding with maximal similarity (Akata et al., 2015; Kodirov et al., 2017). While they can handle well the case where there is an indefinite number of classes during test time, such approaches suffer from extra computation cost at inference time since they need to traverse all the seen data points. Moreover, these models often rely on a pre-trained feature extractor for input, which is usually fixed and cannot be further adapted for the unseen classes (Akata et al., 2015; Kodirov et al., 2017; Elhoseiny et al., 2013). While there is a handful of work that tries to modify the parameters in-place during test time, they either rely on shallow language representation with limited expressiveness (Lei Ba et al., 2015) or require a significant amount of textual descriptions per class to train their model (Reed et al., 2016), both of which are not ideal. In our work, we aim to learn a model that can synthesize parameters for entire neural networks to adapt to the new tasks using short descriptions of the object classes. This leads to better metadata efficiency of our proposed method. Dynamic Parameter Generation As mentioned before, N 3 dynamically generates classification models for designated classes. Dynamic parameter generation has been explored in the context of generating recurrent cells at different time-steps of RNNs (Ha et al., 2016), constructing intermediate linear models for interpretability inside neural networks (Al-Shedivat et al., 2017), and contextual parameter generation for different language pairs in multilingual machine translation (Platanios et al., 2018). As mentioned in Section 2, some zero-shot learning methods can also be viewed as generating classifier parameters (Lei Ba et al., 2015; Elhoseiny et al., 2013). However, many of the previous work directly or indirectly mentions the challenge of memory and computation complexity after all, the output of the parameter generation model are large matrices that are excessively high dimensional. To tackle this issue, previous work either only generate very simple linear layers (Lei Ba et al., 2015; Elhoseiny et al., 2013; Al-Shedivat et al., 2017), or impose low-rank constraints on the weights to mitigate the memory issues (Ha et al., 2016). In our work, we utilize the architecture of sequence-to-sequence models and treat the weight matrices to be generated as a sequence of vectors. This allows parameter generation for entire neural networks with little memory bottleneck. In this section, we describe our approach for synthesizing task-specific neural networks to recognize new objects with their natural language descriptions and a generic pre-trained model. We denote a list of natural language descriptions for K objects, each containing L tokens as D = { d k,l } k = K, l = L k =1 , l =1 . We denote a pre-trained model with parameters  as F (  ; ) , so that for a set of images X , F ( X ; ) produces the classification prediction Y . We can now precisely formulate our problem as follows: given a pre-trained classification model, F (  ; ) , and the natural language description of K object classes, D , adapt the original parameters  to the specialized parameters  (cid:48) so that the fine-tuned F (  ;  (cid:48) ) model accurately classifies the K objects described in D . Our method draws inspiration from transfer learning, which is often employed when the training dataset is small. Transfer learning entails training a neural network from a generic pre-trained model to one that solves a new, often task-specific problem. Thus, we can similarly formulate N 3 to synthesize task-specific model parameters by adapting existing ones in the generic pre-trained model with the guidance of natural language task descriptions. While transfer learning updates the pre-trained parameters using signals derived from task-specific training data, N 3 relies only on language descriptions to achieve the same objective. Concretely, the adapted parameters  (cid:48) are computed as follows:  (cid:48) =  +   ( D ; ) (1) where (  ; ) is a function with parameter  , mapping natural language descriptions to parameter adaptations for all parameters in the pre-trained model. Since the transfer learning process often proceeds with a tiny learning rate to restrict the effect of fine-tuning, we introduced a trainable scaling factor  to similarly regulate the effect of parameter adaptation. The initial value of  is a hyper-parameter. In our experiments, we used an initial  value of 1 e  3 to mimic the effect of using a small learning rate for transfer learning. The specific value of 1 e  3 is derived from the default learning rate used in the PyTorch transfer learning tutorial (Chilamkurthy). In a later section, we evaluate the necessity of this scaling factor as well as the effect of a range of initial values of  . The mapping  from natural language descriptions to parameter adaptations is particularly high-dimensional. Constructing  with the transformer block (Vaswani et al., 2017) is not straight-forward for our scenario because it requires O ( N 2 ) size of memory where N is the length of the input/output sequence and our N = K  L can be prohibitively large. Thus, to reduce the memory consumption of  , its attention span must be restricted to a small but semantically relevant subset of all input elements. To this end, we designed  to be a two-level hierarchy of transformer blocks, as illustrated in 2. The first level of transformers, named the Tokens2Label Encoder , encodes the natural language descriptions of each object class to a label embedding vector. Intuitively, this level summarizes the described visual features of an ob-Tokens2Label Encoder Labels2AdaptationEncoder { A.D.1 ject class to a single, fixed-sized embedding vector. Thus, attention in this level has a maximum span of L spanning all tokens in each description. The second level, named the Labels2Adaptation Encoder-Decoder , encodes the sequence of label embedding vectors and decodes them into parameter adaptations of multiple layers. This level of transformer blocks examines the characteristics of all encoded object classes and determines how to adapt the pre-trained model parameters to classify images corresponding to these object labels. In this level, the model attention has a maximum span of K spanning all the object classes. Moreover, due to the sheer number of layers in state-of-the-art CNN models, we initialize layer-specific adaptation decoders for each layer and decode from shared hidden states encoded by the adaptation encoder. Only through these measures can we materialize the high-dimensional mapping  under reasonable memory constraints. Putting everything together, for a pre-trained network with parameter  , N 3 applies the mapping  to input descriptions to generate a parameter adaptation  with the same shape as  . The adaptation  is multiplied with a trainable scaling factor  to control its impact on the pre-trained model. The scaled parameter adaptation    is then combined with its pre-trained counterpart  via point-wise addition. The mapping  from object descriptions D to  contains two parts. The Tokens2Label Encoder transforms the set of tokens contained in K entries of object class descriptions, each with length up to L , denoted as { d k,l } , to a set of K object label embedding vectors { c k } Kk =1 : c k = Encoder T2L ( d k, 1: L ) , k = 1 , ..., K (2) Subsequently, Labels2Adaptation Encoder-Decoders translate the object label embedding vectors into a parameter adaptation matrix  . Parameters in a typical neural network often have more than two-dimensions, but for simplicity in our setup, they are always viewed as a two-dimensional matrix consisting of a sequence of parameter columns. 1 Viewing the object-label embeddings { c i } ki =1 as an input sequence and the parameter adaptation  as an output sequence of M columns {   m } Mm =1 , Labels2Adaptation Encoder-Decoders can be expressed as: h 1: K = Encoder L2A ( c 1: K ) (3)   m = Decoder mL2A ( h 1: K ) , m = 1 , ..., M (4)  = Concat ([  1 ;   2 ; ... ;   M ]) (5) Finally, pre-trained parameter  is adapted to  (cid:48) using the following equation, with  being a trainable scaling factor:  (cid:48) =  +    (6) 3.3 Training Methodology We formulate the training of N 3 as the following optimization problem. Optimal parameters  for N 3 model (  ; ) should map a list of language descriptions for K class objects D = {D 1 ,    , DK } to parameter adaptations  = ( D ; ) , such that the cross entropy loss between ground truth label Y and model prediction F ( X ;  , ( D ; )) is minimized for all image-label pairs ( X, Y ) in the training set. Thus, to train N 3 on a image dataset I with labels L , class descriptions D and a pre-trained model F to produce a K -way classification model, we first draw meta-batches of K class labels LK  L . Then, a subset of the image dataset ILK and description dataset DLK corresponding to the drawn labels LK are constructed. We then draw mini-batches of images B and ground-truth labels Y 1 For instance, the parameter of a convolutional layer W of shape [ K, C, kH, kW ] where K is the number of output channels, C the number of input channels, kH , kW the kernel height and width, is viewed as a sequence of K parameter columns, with a column size of C  kH  kW . from ILK . For each mini-batch B , distinct parameter adaptations  are generated by evaluating (  ; ) at DL k . Batch loss is then calculated as 1 |B| (cid:80) i (cid:96) ( Y i , F ( X i ;  , )) where (cid:96) (  ,  ) refers to cross-entropy loss. Since the meta-model (  ; ) and the pre-trained model F are fully differentiable, gradients can be propagated back to meta-models to optimize meta-model parameters  . We evaluate N 3 by comparing its efficacy in solving natural language guided zero-shot learning problems with prior state-of-the-art methods. In this section, we introduce our training method, datasets and evaluation protocols. To evaluate the N 3 , we select 4 standard zero-shot image classification datasets and collected natural language descriptions for their object classes. Caltech-UCSD-Birds 200-2011 (CUB) (Wah et al., 2011) contains images of 200 species of birds. Each species of bird forms its own class label. In total, there are 11,788 images in this dataset. Animal with Attributes (AWA) (Lampert et al., 2014; Xian et al., 2017) is another dataset to evaluate zero-shot classification methods. It consists of 50 classes of animals with a total of 37322 images. North America's Birds (NAB) is a dataset used by prior state-of-the-art methods related to our task. Following the established practices (Zhu et al., 2017; Elhoseiny et al., 2017), we consolidated the class labels into 404 distinct bird species. The consolidation process combines closely related labels (e.g., American Kestrel (Female, immature)' and American Kestrel (Adult male)') into a single label (e.g., American Kestrel'). We end up with 48,000 images of 404 classes of bird-species. Flowers-Species (FS) is a dataset we built based on Oxford Flowers (Nilsback and Zisserman, 2006), another commonly used zero-shot dataset. The original contains label categories that are a mixture of species and genera. Some genus includes thousands of species, yet the dataset examples only cover a fraction of them. Such mismatch creates biases in the dataset that fundamentally cannot be addressed through learning from external descriptions. This hence undermines its utility as a test of our proposed method: for instance, when N 3 is asked to generate classifier to decide whether an object is of label anthurium, which is a genus of around 1000 species of varying visual appearance, the efficacy of our generated model can only be evaluated based on a representative samples that cover most of the species within the genus an-thurium. However, the dataset only contains a tiny number of (i.e., 105) correlated (species-wise) samples, making such evaluation neither comprehensive nor conclusive in the context of our task objective and may introduce unexpected noise in evaluation results. Therefore, we decided to filter out the genera from the original Oxford Flowers dataset, leaving only the species as class labels, as an effort towards homogenizing the sample spaces implied by the class labels and the image dataset. This leaves us with 55 classes and 3545 images. For each dataset, we collect language descriptions for object classes from websites like Wikipedia. To collect language descriptions from Wikipedia, we use the python package Wikipedia (Goldsmith) to access structured representation of Wikipedia pages, and extract section content under Description to be used as object class descriptions. If no Wikipedia entry exists for a specific object class, we resort to manually searching for the object class description on Google. Furthermore, we truncate these textual excerpts to a maximum length of 512 tokens to avoid excessively long descriptions and the accompanying computational issues. Recent study showed that previous zero-shot learning evaluation protocols are inadequate and proposed a set of rigorous evaluation protocols for attribute-based zero-shot learning methods (Xian et al., 2017). Although both our tasks and datasets differ, we nevertheless followed Xian et al.'s guiding principles of their Rigorous Protocol and developed our evaluation protocol:  Similar to Rigorous Protocol , we used two meta-splits Standard Splits (SS) and Proposed Splits (PS) to evaluate all methods; the Standard Splits are established meta-splits and Proposed Splits are meta-splits that guarantees the exclusion of ImageNet-1K classes from the test set. Due to class imbalance, Rigorous Protocol proposes to use per-class averaged accuracy for more meaningful evaluation. Thus, to evaluate meta-model on a meta-split containing Datasets Total Standard Split/Proposed Split Training Validation Testing CUB (Wah et al., 2011) 200 100 50 50 AWA2 (Zhu et al., 2017; Elhoseiny et al., 2017) 50 30 10 10 NAB (Zhu et al., 2017) 404 324 40 40 FS (Nilsback and Zisserman, 2006) 55 35 10 10 Table 1: Number of Class Labels in Our Meta-Split (both Standard Split and Proposed Split). averaged accuracy as shown in Equation 7. Acc C = 1 | C | (cid:88) c  C #correctly predicted samples in c #total samples in c (7)  Unlike Rigorous Protocol which uses ResNet-101 model, we use ResNet-18 as our pretrained model; such choice helps reduce the output dimensionality of N 3 by reducing the number of parameters N 3 adapts. Our method is unique in that permutations of the classes belonging to the same meta-split count as distinct tasks and therefore, to account for variations, we test our models on 10 different permutations of test set classes and report the medium value of the relevant evaluation metric. PDCNN PDCNN (Lei Ba et al., 2015) is the most relevant prior method as it use the natural language descriptions of class labels to generate classification layers capable of distinguishing between objects described. Note that PDCNN is distinct from our work in that it dynamically generates fully connected layers and/or additional convolutional layers to be appended to a pre-trained deep neural network (VGG-19) whilst ours generates parameter adaptations to be combined directly with all existing layers within deep neural network models, effectively fine-tuning the pre-trained model. We compare with two variants of PDCNN, with PDCNNFC generating a fully connected layer only for classification and PDCNN FC+Conv producing an additional convolutional layer to help with classification. To make our works comparable, we replaced the TF-IDF feature extractor in PDCNN with a BERT-based document embedding (specifi-cally, a BERT token embedding followed by max-pooling) and changed the pre-trained model from VGG-19 to ResNet-18. MEGAZSL Due to the scarcity of prior work leveraging natural language descriptions for zero-shot classifications in a metadata-efficient way, we also adapted less metadata-efficient methods to function with stricter metadata-efficiency requirements. Specifically, ZSLPP (Elhoseiny et al., 2017), GAZSL (Zhu et al., 2017) and CorrectionNetwork (Hu et al., 2019) all utilize natural language object class descriptions to produce classifiers capable of distinguishing between images belonging to unseen categories during training. However, all of them require significantly more metadata: specifically, parts annotations of each sample image are used to provide extra supervision of the training procedure. Among these methods, GAZSL (Zhu et al., 2017) stands out as the most cited work; therefore we adapted the code released by its authors to learn from only natural language metadata, without using parts annotations; to distinguish our modified version from the original, we will refer to our modified version as MEGAZSL ( M etadataE fficient GAZSL). To make our works comparable, we also updated its language representation from TF-IDF to BERT-based ones and used ResNet-18 as the image feature embedding module. It is worth noting the CorrectionNet(Hu et al., 2019) is orthogonal to our work as it is designed to improve any existing zero-shot classification task modules, and in its original setup, GAZSL (Zhu et al., 2017) was used as the main task module, which we do include in our experimental comparison. For all experiments, hyper-parameters are tuned Method CUB-50 AWA2-10 NAB-40 FS-10 SS PS SS PS SS PS SS PS PDCNNFC (Lei Ba et al., 2015) 6.1 6.1 12.8 18.4 6.7 7.4 13.1 11.9 PDCNN FC+CONV (Lei Ba et al., 2015) 7.5 6.5 22.6 17.2 8.9 5.6 7.7 13.6 MEGAZSL(Zhu et al., 2017) 2.9 1.8 14.0 10.4 2.8 3.7 10.3 12.3 N 3 (Proposed) 17.6 9.5 34.0 37.5 14.1 20.7 16.4 17.6 Table 2: Zero-Shot Classification Accuracy (Defined by Eq. 7) On Various Datasets/Meta-Splits Combinations; number of classification labels used in the test set are recorded next to the name of each dataset. with the same exact algorithms (random search) and for the same number of runs (10). In this section, we compare N 3 method with prior work on 4 zero-shot image classification benchmark datasets. Furthermore, we provide a simple approach to help interpret what part of the language input is being taken as evidence by the synthesized visual classifier to make predictions. We then show through ablation studies the importance of adaptation of all parameters in the pre-trained network, the necessity of pre-trained networks, and the effect of language representation choices on the success of N 3 . We report performance in per-class averaged accuracy, as shown in Table 2. In these experiments, we standardized the language representations, dataset-splits, and other factors orthogonal to our model design to ensure fairness of comparison. We also include experimental results comparing N 3 to models in their respective original settings in the Appendix. From Table 2, we can clearly observe that N 3 outperforms all competing methods by a significant margin on all 8 dataset/meta-split combinations. Noticing the large performance gap between MEGAZSL and the original GAZSL, we performed additional investigations to pinpoint the cause: we reproduced one of their experiments (on CUB dataset with SCE meta-split), and replaced their TF-IDF module with BERT document embedding module. The modified module performed noticeably better (from 10.3% to 11.3%); however, when we replaced its image feature embedding module, which is trained with additional parts annotations of bird images, the performance dropped significantly (from 11.3% to 3.3%), confirming our conjecture that such methods cannot be easily adapted to work without extra supervision in the form of additional data annotation. In this section, we explore how N 3 model architecture can help interpret the predictions made by the adapted model. Specifically, the design of N 3 is unique in that it examines all object class descriptions in order to adapt neural network parameters. This means that N 3 can adapt neural networks to seek both positive and negative evidence for an image to be classified. Naturally, we want to understand how the model is using these object class descriptions. In Figure. 3, we present our findings by visualizing the magnitude of ED ij where E is the loss value computed on a test example (in our experiment, this example is correctly predicted to be an Acadian Flycatcher) and D ij is the BERT representation of the j -th word in the i -th object class description. We present two patterns in the data that are indicative of the model behavior: Top Positive Evidence : top positive evidence is identified as tokens in the description of the ground truth label with large gradients. Intuitively, these tokens are top supporting evidence that encourages the prediction of the correct label. We locate them by ranking all tokens in the description of the ground truth label by their magnitude of gradients and take the top few. Top Negative Evidence : top negative evidence is identified as tokens with large gradients descriptions of the negative labels that also has a low predicted probability. Intuitively, these tokens are the keywords deemed as important evidence when rejecting to predict a label. We locate such evidence by first ranking the descriptions by the ratio between their largest token gradients and the softmax probability of the corresponding label. Then, within the set of class descriptions with the largest aforementioned ratios, we locate the tokens with the largest gradients. Examples for both types of keyword evidence are presented in 3 with their context. Several observations can be drawn about how N 3 uses textual descriptions to make classification decisions. Firstly, we can observe that distinguishing features described with keywords such as olive, darker, yellow and grey are used to support both positive and negative identifications, which support our conjecture that N 3 examines descriptions from all object class descriptions to make classification decisions, sometimes employing the process of elimination. Secondly, we can observe that some label (or label word piece) like flicker and tern is used as evidence to support or refute a classification decision, which seems to suggest that some task-specific knowledge is learned and employed in the language representations. To account for the improved accuracy of models synthesized with whole-model parameter adaptations, we performed additional analysis to understand the features models are utilizing when making predictions. Two variants of N 3 meta-models sharing the same set of hyper-parameters are trained on CUB-50 with the standard split. One is allowed to adapt parameters of every layer whilst the other is ablated only to generate the classification layer. Saliency map (as described in (Si-Figure 4: Saliency map showing magnitude of gradients of loss w.r.t. every pixel in the input image. Left: original image. Middle: saliency map from a model with every layer adapted by N 3 . Right: saliency map from a model where N 3 is restricted to act on only its fc layer. Purple indicates small values while blue/green indicates large values. monyan et al., 2014)) visualizing the importance of each pixel w.r.t. predictions is plotted as a heatmap in Figure. 4. Clearly, fully adapted models show a greater concentration on task-specific regions. To adapt generic pre-trained model parameters to task-specific ones, N 3 mixes the pre-trained model parameters with a set of generated model parameter adaptations, using a trainable mixing ratio as illustrated in Equation 6. It is natural to question whether the pre-trained parameters are necessary. In other words, can N 3 generate parameters for a task-specific model from scratch and achieve reasonable classification accuracy? To study the importance of pre-trained model parameters, we exinitial  Acc@1 0.999 0.001 16.2 % 0.99 0.01 18.5 % 0.9 0.1 16.2 % 0.5 0.5 6.2 % 0.1 0.9 2.1 % 0.01 0.99 2.3 % Table 3: Comparison of different  and initial  values. periment with a modified version of Equation 6:  (cid:48) =    +    Where  is a fixed scalar weight given to the pretrained model,  is still the trainable scaling factor. The choice of  and the initial value of  affects the extent to which the pre-trained model contribute to the parameter-adapted one. In our main experiments, we have fixed  to be 1 and initialized  to be 1  3 , such choice of value initialization proves to work well, yet it remains unclear to what extent the superior performance of N 3 depends on these two hyperparameters. To answer this question, we decide to vary  and  ; In order to keep the magnitude of parameters relatively stable, we always set the initial value of  to be 1   across different settings here. We trained N 3 to produce a 50-way classification model with varying  , and the resultant zero-shot classification performance are shown in Table 3. Such an ablation experiment demonstrates that setting a small initial  is crucial for performance, yet it is not the smaller the better, re-affirming the crucial role of parameter adaptation in achieving good performance. To understand the importance of pre-trained BERT module, and whether N 3 can be generalized to be used with pre-trained word embedding modules other than BERT, we compared the classification accuracy of models adapted by variants of N 3 using different word embeddings. Concretely, we experimented with BERT (Devlin et al., 2018), ELMo (Peters et al., 2018) and GloVe (Pennington et al., 2014) embeddings. Results are shown in Table 4. While BERT prevails as expected, ELMo seems to under-perform GloVe. We hypothesize that ELMo might have been impacted by unexpectedly long sequence lengths (here each description is a paragraph up to 512 tokens), since LSTM-based models are known to be worse at capturing long-range dependencies. To further investigate, we trained a variant of N 3 , where we limit the context of ELMo to each sentence instead of the entire paragraph. As expected, the performance increases noticeably, but the resulting performance is still not on par with other methods. In contrast, GloVe embeddings worked better since such static embeddings are not affected by the long context windows. In this paper, we have demonstrated that small amount of unstructured natural language descriptions for object classes can provide enough information to fine-tune an entire pretrained neural network to perform classification task on class labels unseen during training. We have achieved state-of-the-art performance for natural language guided zero-shot image classification tasks on 4 public datasets with practical metadata requirements. In addition, we presented in-depth analysis and extensive ablation studies on various aspects of the model functioning mechanism and architecture design, showing the necessity of our design contribution in achieving good results. This material is based upon work partially supported by the National Science Foundation (Awards #1750439 #1722822) and National Institutes of Health. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily re-flect the views of National Science Foundation or National Institutes of Health, and no official endorsement should be inferred.
abstain abstain objective abstain objective abstain objective abstain abstain abstain abstain method objective objective result abstain other abstain abstain abstain objective method method objective objective result result abstain method abstain abstain abstain abstain abstain abstain abstain abstain objective objective abstain abstain abstain abstain abstain method abstain abstain abstain method method method abstain abstain method abstain abstain abstain abstain method abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain other abstain method other abstain abstain other abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain method method abstain method method abstain abstain abstain method method method method method method method method abstain method method abstain method abstain method method abstain method method method method abstain abstain method method method other abstain abstain abstain result result abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result result abstain abstain other abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain objective result objective other other