By describing the features and abstractions of our world, language is a crucial tool for human learning and a promising source of supervision for machine learning models. We use language to improve few-shot visual classification in the underexplored scenario where natural language task descriptions are available during training, but unavailable for novel tasks at test time. Existing models for this setting sample new descriptions at test time and use those to classify images. Instead, we propose language-shaped learning (LSL), an end-to-end model that regularizes visual representations to predict language. LSL is conceptually simpler, more data efficient, and outperforms baselines in two challenging few-shot domains. Humans are powerful and efficient learners partially due to the ability to learn from language (Chopra et al., 2019; Tomasello, 1999). For instance, we can learn about robins not by seeing thousands of examples, but by being told that a robin is a bird with a red belly and brown feathers . This language further shapes the way we view the world, constraining our hypotheses for new concepts: given a new bird (e.g. seagulls ), even without language we know that features like belly and feather color are relevant (Goodman, 1955). In this paper, we guide visual representation learning with language, studying the setting where no language is available at test time , since rich linguistic supervision is often unavailable for new concepts encountered in the wild. How can one best use language in this setting? One option is to just regularize, training representations to predict language descriptions. Another is to exploit the compositional nature of language directly by using it as a bottleneck in a discrete latent variable model. For example, the recent Learning with Latent Language (L3; Andreas et al., 2018) model does both: during training, language is used to classify images; at test time, with no language, descriptions are sampled from a decoder conditioned on the language-shaped image embeddings. Whether the bottleneck or regularization most benefits models like L3 is unclear. We disentangle these effects and propose language-shaped learning (LSL), an end-to-end model that uses visual representations shaped by language (Figure 1), thus avoiding the bottleneck. We find that discrete bottlenecks can hurt performance, especially with limited language data; in contrast, LSL is architecturally simpler, faster, uses language more efficiently, and outperforms L3 and baselines across two few-shot transfer tasks. Language has been shown to assist visual classification in various settings, including traditional visual classification with no transfer (He and Peng, 2017) and with language available at test time in the form of class labels or descriptions for zero-(Frome et al., 2013; Socher et al., 2013) or few-shot (Xing et al., 2019) learning. Unlike past work, we have no language at test time and test tasks differ from training tasks, so language from training cannot be used as additional class information (cf. He and Peng, 2017) or weak supervision for labeling additional in-domain data (cf. Hancock et al., 2018). Our setting can be viewed as an instance of learning using privileged information (LUPI; Vapnik and Vashist, 2009), where richer supervision augments a model only during training. In this framework, learning with attributes and other domain-specific rationales has been tackled extensively (Zaidan et al., 2007; Donahue and Grau-man, 2011; Tokmakov et al., 2019); language less so. Gordo and Larlus (2017) use METEOR scores between captions as a similarity measure for specializing embeddings for image retrieval, but do not directly ground language explanations. Srivastava et al. (2017) explore a supervision setting similar to ours, except in simple text and symbolic domains where descriptions can be easily converted to executable logical forms via semantic parsing. Another line of work studies the generation of natural language explanations for interpretability across language (e.g. entailment; Camburu et al., 2018) and vision (Hendricks et al., 2016, 2018) tasks, but here we examine whether predicting language can actually improve task performance; similar ideas have been explored in text (Rajani et al., 2019) and reinforcement learning (Bahdanau et al., 2019; Goyal et al., 2019) domains. We are interested in settings where language explanations can help learn representations that generalize more efficiently across tasks, especially when training data for each task is scarce and there are many spurious hypotheses consistent with the input. Thus, we study the few-shot (meta-)learning setting, where a model must learn from a set of train tasks, each with limited data, and then generalize to unseen tasks in the same domain. with K examples each: S ( t ) n = { x ( t ) n, 1 , . . . , x ( t ) n,K } . Each task has M query examples Q ( t ) = { ( x ( t ) 1 , y ( t ) 1 ) , . . . , ( x ( t ) M , y ( t ) M ) } . Given the m -th query example x ( t ) m as input, the goal is to predict its class y ( t ) m  { 1 , . . . , N } . After learning from a set of tasks T train , a model is evaluated on unseen tasks T test . While the language approach we propose is applicable to nearly any meta-learning framework, we use prototype networks (Snell et al., 2017), which have a simple but powerful inductive bias for few-shot learning. Prototype networks learn an embedding function f  for examples; the embeddings of the support examples of a class n are averaged to form a class prototype (omitting task ( t ) for clarity): c n = 1 KK (cid:88) k =1 f  ( x n,k ) . Given a query example ( x m , y m ) , we predict class n with probability proportional to some similarity function s between c n and f  ( x m ) : Now assume that during training we have for each class S n a set of J n associated natural language descriptions W n = { w 1 , . . . , w J n } . Each w j should explain the relevant features of S n and need not be associated with individual examples. 1 In Figure 1, we have one description w 1 = ( A , red , . . . , square ) . Our approach is simple: we encourage f  to learn prototypes that can also decode the class language descriptions. Let  c n be the prototype formed by averaging the support and query examples of class n . Then define a language model g  (e.g., a recurrent neural network), which conditioned on 1 If we have language associated with individual examples, we can regularize at the instance-level, essentially learning an image captioner. We did not observe major gains with instance-level supervision (vs class-level) in the tasks explored here, in which case class-level language is preferable, since it is much easier to obtain. There are likely tasks where instance-level supervision is superior, which we leave for future work. c n provides a probability distribution over descriptions g  (  w j |  c n ) with a corresponding natural language loss : LNL ( ,  ) =  N (cid:88) n =1 J n (cid:88) j =1 log g  ( w j |  c n ) , (4) i.e. the total negative log-likelihood of the class descriptions across all classes in the task. Since LNL depends on parameters  through the prototype  c n , this objective should encourage our model to better represent the features expressed in language. Now we jointly minimize both losses: arg min , [ LCLS (  ) +  NLLNL ( ,  )] , (5) where the hyperparameter  NL controls the weight of the natural language loss. At test time, we simply discard g  and use f  to classify. We call our approach language-shaped learning (LSL; Figure 1). L3 (Andreas et al., 2018) has the same basic components of LSL, but instead defines the concepts c n to be embeddings of the language descriptions themselves, generated by an additional recurrent neural network (RNN) encoder h  : c n = h  ( w n ) . During training, the ground-truth description is used for classification, while g  is trained to produce the description; at test time, L3 samples candidate descriptions  w n from g  , keeping the description most similar to the images in the support set according to the similarity function s (Figure 1). Compared to L3, LSL is simpler since it (1) does not require the additional embedding module h  and (2) does not need the test-time language sampling procedure. 2 This also makes LSL much faster to run than L3 in practice: without the language machinery, LSL is up to 50x faster during inference in our experiments. Here we describe our two tasks and models. For each task, we evaluate LSL, L3, and a prototype network baseline trained without language (Meta; Figure 1). For full details, see Appendix A. 2 LSL is similar to the Meta+Joint model of Andreas et al. (2018), which did not improve over baseline. However, they used separate encoders for the support and query examples, with only the support encoder trained to predict language, resulting in overfitting of the query encoder. ShapeWorld. First, we use the ShapeWorld (Kuhnle and Copestake, 2017) dataset used by Andreas et al. (2018), which consists of 9000 training, 1000 validation, and 4000 test tasks (Figure 2). 3 Each task contains a single support set of K = 4 images representing a visual concept with an associated (artificial) English language description, generated with a minimal recursion semantics representation of the concept (Copestake et al., 2016). Each concept is a spatial relation between two objects, each object optionally qualified by color and/or shape, with 2-3 distractor shapes present. The task is to predict whether a query image x belongs to the concept. For ease of comparison, we report results with models identical to Andreas et al. (2018), where f  is the final convolutional layer of a fixed ImageNet-pretrained VGG-16 (Simonyan and Zisserman, 2015) fed through two fully-connected layers: f  ( x ) = FC(ReLU(FC(VGG-16( x )))) . However, because fixed ImageNet representations may not be the most appropriate choice for artificial data, we also run experiments with convolutional networks trained from scratch: either the 4-layer convolutional backbone used in much of the few-shot literature (Chen et al., 2019), as used in the Birds experiments we describe next, or a deeper ResNet-18 (He et al., 2016). This is a special binary case of the few-shot learning framework, with a single positive support class S and prototype c . Thus, we define the similarity function to be the sigmoid function s ( a, b ) =  ( a  b ) and the positive prediction P ( y = 1 | x ) = s ( f  ( x ) , c ) . g  is a 512-dimensional gated recurrent unit (GRU) RNN (Cho et al., 2014) trained with teacher forcing. Through a grid search on the validation set, we set  NL = 20 . Birds. To see if LSL can scale to more realistic scenarios, we use the Caltech-UCSD Birds dataset (Wah et al., 2011), which contains 200 bird species, each with 4060 images, split into 100 train, 50 validation, and 50 test classes. During training, tasks are sampled dynamically by selecting N classes from the 100 train classes. K support and 16 query examples are then sampled from each class (sim-ilarly for val and test). For language, we use the descriptions collected by Reed et al. (2016), where 3 This is a larger version with 4x as many test tasks for more stable confidence intervals (see Appendix A). AMT crowdworkers were asked to describe individual images of birds in detail, without reference to the species (Figure 2). While 10 English descriptions per image are available, we assume a more realistic scenario where we have much less language available only at the class level: removing associations between images and their descriptions, we aggregate D descriptions for each class, and for each K -shot training task we sample K descriptions from each class n to use as descriptions W n . This makes learning especially challenging for LSL due to noise from captions that describe features only applicable to individual images. Despite this, we found improvements with as few as D = 20 descriptions per class, which we report as our main results, but also vary D to see how efficiently the models use language. We evaluate on the N = 5 -way, K = 1 -shot setting, and as f  use the 4-layer convolutional backbone proposed by Chen et al. (2019). Here we use a learned bilinear similarity function, s ( a, b ) = a (cid:62) W b , where W is learned jointly with the model. g  is a 200-dimensional GRU, and with another grid search we set  NL = 5 . Results are in Table 1. For ShapeWorld, LSL outperforms the meta-learning baseline (Meta) by 6.7%, and does at least as well as L3; Table 2 shows similar trends when f  is trained from scratch. For Birds, LSL has a smaller but still significant 3.3% increase over Meta, while L3 drops below baseline. Furthermore, LSL uses language more efficiently: Figure 3 shows Birds performance as the captions per class D increases from 1 (100 total) to 60 (6000 total). LSL benefits from a remarkably small number of captions, with limited gains past 20; in contrast, L3 requires much more language to 50 52 54 56 58 60 1 5 10 20 30 40 50 60 D descriptions/class B i r d s A cc u r a cy Model LSLL3 Figure 3: Varying the descriptions per class, D , for Birds. In the low-data regime, L3's lower performance is unsurprising, since it must generate language at test time, which is difficult with so little data. Example output from the L3 decoder in Figure 4 highlights this fact: the language looks reasonable in some cases, but in others has factual errors ( dark gray bird; black pointed beak ) and fluency issues. These results suggest that any benefit of L3 is likely due to the regularizing effect that language has on its embedding model f  , which has been trained to predict language for test-time inference; in fact, the discrete bottleneck actually hurts in some settings. By using only the regularized visual representations and not relying exclusively on the generated language, LSL is the simpler, more efficient, and overall superior model. To identify which aspects of language are most helpful, in Figure 5 we examine LSL performance under ablated language supervision: (1) keeping only a list of common color words, (2) filtering out color words, (3) shuffling the words in each caption, and (4) shuffling the captions across tasks (see Figure 6 for examples). We find that while the benefits of color/no-color language varies across tasks, neither component provides the benefit of complete language, demonstrating that LSL leverages both colors and other attributes (e.g. size, shape) described in language. Word order is important for Birds but surprisingly unimportant for ShapeWorld, suggesting that even with decoupled colors and shapes, the model can often infer the correct relation from the shapes that consistently appear in the examples. Finally, when captions are shuffled across tasks, LSL for Birds does no worse than Meta, while ShapeWorld suffers, suggesting that language is more important for ShapeWorld than for the fine-grained, attribute-based Birds task. We presented LSL, a few-shot visual recognition model that is regularized with language descriptions during training. LSL outperforms baselines across two tasks and uses language supervision more efficiently than L3. We find that if a model is trained to expose the features and abstractions in language, a linguistic bottleneck on top of these Birds ShapeWorld a cyan pentagon is to the right of a magenta shape cyan magenta a pentagon is to the right of a shape shape right the is a pentagon a of cyan to magenta a green square is below a triangle The bird has a white underbelly, black feathers in the wings, a large wingspan, and a white beak. language-shaped representations is unnecessary, at least for the kinds of visual tasks explored here. The line between language and sufficiently rich attributes and rationales is blurry, and recent work (Tokmakov et al., 2019) suggests that similar performance gains can likely be observed by regularizing with attributes. However, unlike attributes, language is (1) a more natural medium for annotators, (2) does not require preconceived restrictions on the kinds of features relevant to the task, and (3) is abundant in unsupervised forms. This makes shaping representations with language a promising and easily accessible way to improve the generalization of vision models in low-data settings. We thank Pang Wei Koh, Sebastian Schuster, and Dan Iter for helpful discussions and feedback, Mike Wu and Jacob Andreas for discussions and code, and our anonymous reviewers for insightful comments. This work was supported by an NSF Graduate Research Fellowship for JM, a SAIL-Toyota Research Award, and the Office of Naval Research grant ONR MURI N00014-16-1-2007. Toyota Research Institute (TRI) provided funds to assist the authors with their research but this article solely reflects the opinions and conclusions of its authors and not TRI or any other Toyota entity. Code, data, and experiments are available at https: //github.com/jayelm/lsl and on CodaLab at https://bit.ly/lsl_acl20 .
abstain objective abstain objective abstain abstain result objective objective abstain abstain abstain abstain abstain objective objective other abstain abstain other other other other abstain method abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain method abstain abstain other abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain method abstain result abstain abstain abstain abstain other other other other