summaries: This paper focuses on improving zero-shot classification by reducing the bias of the classifier towards seen classes. The bias occurs since the embedding is trained with visual examples from the seen classes, while using only the attribute information from unseen classes for testing. Authors propose an isometric propagation network that build a graph in both visual and semantic space, performs some steps of propagation, and then uses the updated prototypes for training a classifier. They use attention to construct the graph and also use attention to regularize the graph edges between the two spaces to be isometric. Authors also propose to use an episodic training method to improve learning.  In this paper Zero Shot Classification is studied using prototypes. Each class is represented with a visual and semantic prototype, and at test time compared to a visual example + prototype for a(n unseen) test class. The most similar test class is chosen. In this work a novel method is proposed to construct the prototypes, which are trained in an episode learning setting. On various benchmarks the proposed method performs better than existing method for the generalized zero-shot classification task (seen + unseen) test classes. The authors propose a novel computational pipeline to tackle a well-known problem in zero-shot learning: although multiple visual instances are available for the classes and categories to be recognized, one and only one semantic embedding is available to describe the classes/categories while using side information like attributes or relevant textual information. To cope with that problem, authors learn visual and semantic prototypes which are then adopted to perform gradient descent over a graph in which the topological relationship among similar/dissimilar classes are preserved. In the experimental validation, the proposed method shows its superiority among a number of prior methods in zero-shot learning, including discriminative and generative methods. 
abstractText: Zero-shot learning (ZSL) aims to classify images of an unseen class only based on a few attributes describing that class but no access to any training sample. A popular strategy is to learn a mapping between the semantic space of class attributes and the visual space of images based on the seen classes and their data. Thus, an unseen class image can be ideally mapped to its corresponding class attributes. The key challenge is how to align the representations in the two spaces. For most ZSL settings, the attributes for each seen/unseen class are only represented by a vector while the seen-class data provide much more information. Thus, the imbalanced supervision from the semantic and the visual space can make the learned mapping easily overfitting to the seen classes. To resolve this problem, we propose Isometric Propagation Network (IPN), which learns to strengthen the relation between classes within each space and align the class dependency in the two spaces. Specifically, IPN learns to propagate the class representations on an auto-generated graph within each space. In contrast to only aligning the resulted static representation, we regularize the two dynamic propagation procedures to be isometric in terms of the two graphs’ edge weights per step by minimizing a consistency loss between them. IPN achieves state-of-the-art performance on three popular ZSL benchmarks. To evaluate the generalization capability of IPN, we further build two larger benchmarks with more diverse unseen classes, and demonstrate the advantages of IPN on them.
authors.name: Lu Liu
authors.name: Tianyi Zhou
authors.name: Guodong Long
authors.name: Jing Jiang
authors.name: Xuanyi Dong
authors.name: Chengqi Zhang
references.authors: Zeynep Akata Florent Perronnin Zaid Harchaoui Cordelia Schmid
references.title: Label-embedding for image classification
references.venue: TPAMI, 2015a. doi: 10.1109/TPAMI.2015.2487986
references.authors: Zeynep Akata Scott Reed Daniel Walter Honglak Lee Bernt Schiele
references.title: Evaluation of output embeddings for fine-grained image classification
references.venue: In CVPR,
references.authors: Yannick Le Cacheux Herve Le Borgne Michel Crucianu
references.title: Modeling inter and intra-class relations in the triplet loss for zero-shot learning
references.authors: Soravit Changpinyo Wei-Lun Chao Boqing Gong Fei Sha
references.title: Synthesized classifiers for zeroshot learning
references.authors: Soravit Changpinyo Wei-Lun Chao Fei Sha
references.title: Predicting visual exemplars of unseen classes for zero-shot learning
references.venue: In ICCV,
references.authors: Haoang Chi Feng Liu Wenjing Yang Long Lan Tongliang Liu Gang Niu Bo Han
references.title: Meta discovery: Learning to discover novel classes given very limited data, 2021
references.authors: Kaize Ding Jianling Wang Jundong Li Kai Shu Chenghao Liu Huan Liu
references.title: Graph prototypical networks for few-shot learning on attributed networks
references.venue: In Proceedings of the 29th ACM International Conference on Information & Knowledge Management,
references.authors: Kaize Ding Qinghai Zhou Hanghang Tong huan Liu
references.title: Few-shot network anomaly detection via cross-network meta-learning
references.venue: In Proceedings of The Web Conference
references.authors: Xuanyi Dong Yi Yang
references.title: Searching for a robust neural architecture in four gpu hours
references.authors: Xuanyi Dong Lu Liu Katarzyna Musial Bogdan Gabrys
references.title: NATS-Bench: Benchmarking nas algorithms for architecture topology and size
references.authors: Hehe Fan Ping Liu Mingliang Xu Yi Yang
references.title: Unsupervised visual representation learning via dual-level progressive similar instance selection. TCYB, 2021
references.authors: Ali Farhadi Ian Endres Derek Hoiem David Forsyth
references.title: Describing objects by their attributes
references.venue: In CVPR,
references.authors: Chelsea Finn Pieter Abbeel Sergey Levine
references.title: Model-agnostic meta-learning for fast adaptation of deep networks
references.venue: In ICML,
references.authors: Andrea Frome Greg S Corrado Jon Shlens Samy Bengio Jeff Dean Tomas Mikolov
references.title: Devise: A deep visual-semantic embedding model
references.venue: NeurIPS,
references.authors: Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville Yoshua Bengio
references.title: Generative adversarial nets
references.venue: NeurIPS,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.authors: He Huang Changhu Wang Philip S Yu Chang-Dong Wang
references.title: Generative dual adversarial network for generalized zero-shot learning
references.authors: Huajie Jiang Ruiping Wang Shiguang Shan Xilin Chen
references.title: Learning class prototypes via structure alignment for zero-shot recognition
references.venue: In ECCV,
references.authors: Michael Kampffmeyer Yinbo Chen Xiaodan Liang Hao Wang Yujia Zhang Eric P Xing
references.title: Rethinking knowledge graph propagation for zero-shot learning
references.authors: Diederik P Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: In ICLR,
references.authors: Elyor Kodirov Tao Xiang Shaogang Gong
references.title: Semantic autoencoder for zero-shot learning
references.venue: In CVPR,
references.authors: Christoph H Lampert Hannes Nickisch Stefan Harmeling
references.title: Attribute-based classification for zero-shot visual object categorization
references.authors: Hugo Larochelle Dumitru Erhan Yoshua Bengio
references.title: Zero-data learning of new tasks
references.venue: In IJCAI,
references.authors: Jingjing Li Mengmeng Jin Ke Lu Zhengming Ding Lei Zhu Zi Huang
references.title: Leveraging the invariant side of generative zero-shot learning
references.venue: In CVPR,
references.authors: Kai Li Martin Renqiang Min Yun Fu
references.title: Rethinking zero-shot learning: A conditional visual classification perspective
references.venue: In CVPR,
references.authors: Yanan Li Donghui Wang Huanhang Hu Yuetan Lin Yueting Zhuang
references.title: Zero-shot recognition using dual visual-semantic mapping paths
references.authors: Feng Liu Wenkai Xu Jie Lu Guangquan Zhang Arthur Gretton Danica J. Sutherland
references.title: Learning deep kernels for non-parametric two-sample tests
references.venue: In ICML,
references.authors: Lu Liu Tianyi Zhou Guodong Long Jing Jiang Lina Yao Chengqi Zhang
references.title: Prototype propagation networks (PPN) for weakly-supervised few-shot learning on category graph
references.venue: In IJCAI,
references.authors: Lu Liu Tianyi Zhou Guodong Long Jing Jiang Chengqi Zhang
references.title: Learning to propagate for graph meta-learning
references.venue: In NeurIPS,
references.authors: Shichen Liu Mingsheng Long Jianmin Wang Michael I Jordan
references.title: Generalized zero-shot learning with deep calibration network
references.venue: In NeurIPS,
references.authors: Yang Liu Jishun Guo Deng Cai Xiaofei He
references.title: Attribute attention for semantic disambiguation in zero-shot learning
references.venue: In ICCV,
references.authors: Yang Liu Deyan Xie Quanxue Gao Jungong Han Shujian Wang Xinbo Gao
references.title: Graph and autoencoder based feature extraction for zero-shot learning
references.venue: In IJCAI,
references.authors: Yixin Liu Shirui Pan Ming Jin Chuan Zhou Feng Xia Philip S. Yu
references.title: Graph self-supervised learning: A survey, 2021
references.authors: Laurens van der Maaten Geoffrey Hinton
references.title: Visualizing data using t-SNE
references.authors: Mohammad Norouzi Tomas Mikolov Samy Bengio Yoram Singer Jonathon Shlens Andrea Frome Greg S Corrado Jeffrey Dean
references.title: Zero-shot learning by convex combination of semantic embeddings
references.venue: NeurIPS,
references.authors: Jeffrey Pennington Richard Socher Christopher Manning
references.title: Glove: Global vectors for word representation
references.venue: In EMNLP,
references.authors: Sachin Ravi Hugo Larochelle
references.title: Optimization as a model for few-shot learning
references.venue: In ICLR,
references.authors: Mengye Ren Eleni Triantafillou Sachin Ravi Jake Snell Kevin Swersky Joshua B Tenenbaum Hugo Larochelle Richard S Zemel
references.title: Meta-learning for semi-supervised few-shot classification
references.venue: In ICLR,
references.authors: Bernardino Romera-Paredes Philip Torr
references.title: An embarrassingly simple approach to zero-shot learning
references.venue: In ICML,
references.authors: Adam Santoro Sergey Bartunov Matthew Botvinick Daan Wierstra Timothy Lillicrap
references.title: Metalearning with memory-augmented neural networks
references.venue: In ICML,
references.authors: Mert Bulent Sariyildiz Ramazan Gokberk Cinbis
references.title: Gradient matching generative networks for zero-shot learning
references.authors: Jake Snell Kevin Swersky Richard Zemel
references.title: Prototypical networks for few-shot learning
references.venue: In NeurIPS,
references.authors: Richard Socher Milind Ganjoo Christopher D Manning Andrew Ng
references.title: Zero-shot learning through cross-modal transfer
references.venue: In NeurIPS,
references.authors: Robyn Speer Joshua Chin Catherine Havasi
references.title: Conceptnet 5.5: An open multilingual graph of general knowledge
references.venue: arXiv preprint arXiv:1612.03975,
references.authors: Flood Sung Yongxin Yang Li Zhang Tao Xiang Philip HS Torr Timothy M Hospedales
references.title: Learning to compare: Relation network for few-shot learning
references.authors: Bin Tong Chao Wang Martin Klinkigt Yoshiyuki Kobayashi Yuuichi Nonaka
references.title: Hierarchical disentanglement of discriminative latent features for zero-shot learning
references.authors: Xiaolong Wang Yufei Ye Abhinav Gupta
references.title: Zero-shot recognition via semantic embeddings and knowledge graphs
references.authors: Peter Welinder Steve Branson Takeshi Mita Catherine Wah Florian Schroff Serge Belongie Pietro Perona
references.title: Caltech-ucsd birds 200
references.venue: Technical report, California Institute of Technology,
references.authors: Yongqin Xian Christoph H Lampert Bernt Schiele Zeynep Akata
references.title: Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly
references.venue: TPAMI, 2019a. doi: 10.1109/TPAMI
references.authors: Yongqin Xian Saurabh Sharma Bernt Schiele Zeynep Akata
references.title: f-vaegan-d2: A feature generating framework for any-shot learning
references.venue: In CVPR,
references.authors: Guo-Sen Xie Li Liu Xiaobo Jin Fan Zhu Zheng Zhang Jie Qin Yazhou Yao Ling Shao
references.title: Attentive region embedding network for zero-shot learning
references.authors: Guo-Sen Xie Li Liu Fan Zhu Fang Zhao Zheng Zhang Yazhou Yao Jie Qin Ling Shao
references.title: Region graph embedding network for zero-shot learning
references.authors: Fei Zhang Guangming Shi
references.title: Co-representation network for generalized zero-shot learning
references.venue: In ICML,
references.authors: Li Zhang Tao Xiang Shaogang Gong
references.title: Learning a deep embedding model for zero-shot learning
references.authors: Pengkai Zhu Hanxiao Wang Venkatesh Saligrama
references.title: Generalized zero-shot recognition based on visually semantic embedding
references.venue: In CVPR,
references.authors: Yizhe Zhu Jianwen Xie Zhiqiang Tang Xi Peng Ahmed Elgammal
references.title: Semantic-guided multiattention localization for zero-shot learning
references.venue: In NeurIPS,
references.authors: Frome
references.title: who proposed to use the 21K ImageNet dataset for zero-shot evaluation. Here, 2-hops and 3-hops represent that the evaluation is over the classes which are 2/3 hops away from ImageNet-1k. Different from experiments on AWA2, we use N=20 and K=5 for the full ImageNet
sections.heading: 1 INTRODUCTION
sections.text: One primary challenge on the track from artificial intelligence to human-level intelligence is to improve the generalization capability of machine learning models to unseen problems. While most supervised learning methods focus on generalization to unseen data from training task/classes, zeroshot learning (ZSL) (Larochelle et al., 2008; Lampert et al., 2014; Xian et al., 2019a) has a more ambitious goal targeting the generalization to new tasks and unseen classes. In the context of image classification, given images from some training classes, ZSL aims to classify new images from unseen classes with zero training image available. Without training data, it is impossible to directly learn a mapping from the input space to the unseen classes. Hence, recent works introduced learning to map between the semantic space and visual space (Zhu et al., 2019a; Li et al., 2017; Jiang et al., 2018) so that the query image representation and the class representation can be mapped to a shared space for comparison.
  However, learning to align the representation usually leads to overfitting on seen classes (Liu et al., 2018). One of the reasons is that in most zero-shot learning settings, the number of images per class is hundreds while the semantic information provided for one class is only limited to one vector. Thus, the mapping function is easily overfitting to the seen classes when trained using the small set of attributes. The learned model typically has imbalanced performance on samples from the seen classes and unseen classes, i.e., strong when predicting samples from the seen classes but struggles in the prediction for samples in the unseen classes.
  In this paper, we propose “Isometric Propagation Network (IPN)” which learns to dynamically interact between the visual space and the semantic space. Within each space, IPN uses an attention module to generate a category graph, and then perform multiple steps of propagation on the graph.
  In every step, the distribution of the attention scores generated in each space are regularized to be isometric so that implicitly aligns the relationships between classes in two spaces. Due to different motifs of the generated graphs, the regularizer for the isometry between the two distributions are provided with sufficient training supervisions and can potentially generalize better on unseen classes. To get more diverse graph motifs, we also apply episodic training which samples different subset of classes as the nodes for every training episode rather than the whole set of seen classes all the time.
  To evaluate IPN, we compared it with several state-of-the-art ZSL methods on three popular ZSL benchmarks, i.e., AWA2 (Xian et al., 2019a), CUB (Welinder et al., 2010) and aPY (Farhadi et al., 2009). To test the generalization ability on more diverse unseen classes rather than similar classes to the seen classes, e.g., all animals for AWA2 and all birds for CUB, we also evaluate IPN on two new large-scale datasets extracted from tieredImageNet (Ren et al., 2018), i.e., tieredImageNet-Mixed and tieredImageNet-Segregated. IPN consistently achieves state-of-the-art performance in terms of the harmonic mean of the accuracy on unseen and seen classes. We show the ablation study of each component in our model and visualize the final learned representation in each space in Figure 1. It shows the class representations in each space learned by IPN are evenly distributed and connected smoothly between the seen and unseen classes. The two spaces can also reflect each other.
sections.heading: 2 RELATED WORKS
sections.text: Most ZSL methods can be categorized as either discriminative or generative. Discriminative methods generate the classifiers for unseen classes from its attributes by learning a mapping from a input image’s visual features to the associated class’s attributes on training data of seen classes (Frome et al., 2013; Akata et al., 2015a;b; Romera-Paredes & Torr, 2015; Kodirov et al., 2017; Socher et al., 2013; Cacheux et al., 2019; Li et al., 2019b), or by relating similar classes according to their shared properties, e.g., semantic features (Norouzi et al., 2013; Chi et al., 2021) or phantom classes (Changpinyo et al., 2016), etc. In contrast, generative methods employ the recently popular image generation models (Goodfellow et al., 2014) to synthesize images of unseen classes, which reduces ZSL to supervised learning (Huang et al., 2019) but usually increases the training cost and model complexity. In addition, the study of unsupervised learning (Fan et al., 2021; Liu et al., 2021), meta learning (Santoro et al., 2016; Liu et al., 2019b;a; Ding et al., 2020; 2021), or architecture design (Dong et al., 2021; Dong & Yang, 2019) can also be used improve the representation ability of visual or semantic features.
  Alignment between two spaces in ZSL Frome et al. (2013); Socher et al. (2013); Zhang et al. (2017) proposed to transform representations either to the visual space or the semantic space for distance for comparison. Li et al. (2017) introduces how to optimize the semantic space so that the two spaces can be better aligned. Jiang et al. (2018); Liu et al. (2020) tries to map the class representations from two spaces to another common space. Comparatively, our IPN does not align two spaces by feature transformation but by regularizing the isometry of the intermediate output from two propagation model.
  Graph in ZSL Graph structures have already been explored for their benefits to some discriminative ZSL models, e.g., Wang et al. (2018); Kampffmeyer et al. (2019) learn to generate the weights of a fully-connected classifier for a given class based on its semantic features computed on a pre-
  defined semantic graph using a graph convolutional networks (GCN). They require (1) a pre-trained CNN whose last fully-connected layer provides the ground truth of the classifiers ZSL aims to generate; and (2) an existing knowledge graph as the input of GCN. By contrast, our method learns to automatically generate the graph and the class prototypes composing the support set of a distancebased classifier. Hence, our method does not require a pre-trained CNN to provide a ground truth for the classifier. In addition, we train a modified graph attention network (GAN) (rather than a GCN) to generate the prototypes, so it learns both the class adjacency and edges by itself and can provide more accurate modeling of class dependency in scenarios when no knowledge graph is provided. Tong et al. (2019) and Xie et al. (2020) also leverage the idea of graph to transform the feature extracted from the CNN while our graph serves as the paths for propagation. It requires an off-the-shelf graph generated by ConceptNet (Speer et al., 2016), while IPN applies prototype propagation and learns to construct the graph in an end-to-end manner.
  Attention in ZSL The attention module has been applied for zero-shot learning for localization. Liu et al. (2019c) introduces localization on the semantic information and Xie et al. (2019) proposes localization on the visual feature. Zhu et al. (2019b) further extends it to multi-localization guided by semantic. Different from them, our attention serves as generating the weights for propagation.
  Our episodic training strategy is inspired by the meta-learning method from (Santoro et al., 2016). Similar methods have been broadly studied in meta-learning and few-shot learning approaches (Finn et al., 2017; Snell et al., 2017; Sung et al., 2018) but rarely studied for ZSL in previous works. In this paper, we show that training ZSL model as a meta-learner can improve the generalization performance and mitigate overfitting.
sections.heading: 3 PROBLEM SETTING
sections.text: We consider the following setting of ZSL. The training set is composed of data-label pairs (x, y), where x ∈ X is a sample from class y, and y must be one of the seen classes in Yseen. During the test stage, given a test sample x of class y , the ZSL model achieved in training is expected to produce the correct prediction of y. In order to relate the unseen classes in test to the seen classes in training, we are also given the semantic embedding sy of each class y ∈ Yseen ∪ Yunseen. We further assume that seen and unseen classes are disjoint, i.e., Yseen∩Yunseen = ∅. The test samples come from both Yseen and Yunseen in generalized zero-shot learning.
sections.heading: 4 ISOMETRIC PROPAGATION NETWORK
sections.text: In this paper, we bridge the two spaces via class prototypes in both the semantic and visual spaces, i.e., a semantic prototype P sy and a visual prototype P v y for each class y. In order to generate high-quality prototypes, we develop a learnable scheme called “Isometric Propagation Network
  (IPN)” to iteratively refine the prototypes by propagating them on a graph of classes so each prototype is dynamically determined by the samples and attributes of its related classes.
  In this section, we will first introduce the architecture of IPN. An illustration of IPN is given in Figure 2. Then, we will elaborate the episodic training strategy, in which a consistency loss is minimized for isometric alignment between the two spaces.
sections.heading: 4.1 PROTOTYPE INITIALIZATION
sections.text: Given a task T defined on a subset of classes YT with training data DT , IPN first initializes the semantic and visual prototype for each class y ∈ YT . The semantic prototype of class y is transformed from its attributes sy . Specifically, we have drawn on the expert modules from (Zhang & Shi, 2019) to initialize the semantic prototypes P sy [0]. In previous works (Snell et al., 2017; Santoro et al., 2016), the visual prototype of a class is usually computed by averaging the features of images from the class. In IPN, for each seen class y ∈ YT ∩ Yseen, we initialize its visual prototype P vy [0] by
  P vy [0] = W |DTy | ∑
  (xi,yi)∈DTy
  fcnn(xi), D T y , {(xi, yi) ∈ DT : yi = y}, (1)
  where fcnn(·) is a backbone convolutional neural nets (CNN) extracting features from raw images, and W is a linear transformation aiming to change the dimension of the visual prototype to be the same as semantic prototypes. We argue that W is necessary because (1) the visual feature produced by the CNN is usually in high dimension, and reducing its dimension can save significant computations during propagation; (2) keeping the dimensions of the visual and semantic prototypes the same can avoid the dominance of either of them in the model.
  During test, we initialize the visual prototype of each unseen class y ∈ YT∩Yunseen as the weighted sum of its K-nearest classes from Yseen in the semantic space1, i.e.,
  P vy [0] = W · ∑ z∈Ny as(P sy [0],P s z [0]) · P vz , (2)
  where W is the same transformation from Eq. (1), and Ny is the set of the top-K (K is a hyperparameter) seen classes2 with the largest similarities cs(P sy [0],P s z [0]) to class y in semantic space.
sections.heading: 4.2 CATEGORY GRAPH GENERATION
sections.text: Although the propagation needs to be conducted on a category graph, IPN does not rely on any given graph structure as many previous works (Kampffmeyer et al., 2019; Wang et al., 2018) so it has less limitations when applied to more general scenarios. In IPN, we inference the category graph of YT by applying thresholding to the attention scores between the initialized prototypes. In the visual space, the set of edges is defined by
  ET,v = { (y, z) : y, z ∈ YT , cv(P vy [0],P vz [0]) ≥ } , (3)
  where is the threshold. Thereby, the category graph is generated as GT,v = (YT , ET,v). Similarly, in the semantic space, we generate another graph GT,s = (YT , ET,s).
sections.heading: 4.3 ISOMETRIC PROPAGATION
sections.text: Given the initialized prototypes and generated category graphs in the two spaces, we apply the following prototype propagation in each space for τ steps in order to refine the prototype of each class. In the visual space, each propagation step updates
  P vy [t+ 1] = ∑
  z:(y,z)∈ET,v av(P vy [t],P v z [t]) · P vz [t], (4)
  1This only happens in the test since we do not use any information of unseen classes during training. 2Here the notation Ny needs a little overloading since it denotes the set of neighbor classes of y elsewhere
  in the paper.
  In the semantic space, we apply similar propagation steps over classes on graph GT,s by using the attention module associated with as(·, ·). Intuitively, the attention module computes the similarity between the two prototypes. In the visual space, given the prototypes P vy and P v z for class y and z, the attention module first computes the following cosine similarity:
  cv(P vy ,P v z ) = 〈hv(P vy ), hv(P vz )〉 ‖hv(P vy )‖2 · ‖hv(P vz )‖2 , (5)
  where hv(·) is a learnable transformation and ‖ · ‖2 represents the `2 norm. For a class y on a category graph, we can use the attention module to aggregate messages from its neighbor classes Ny . In this case, we need to further normalize the similarities of y to each z ∈ Ny as follows.
  av(P vy ,P v z ) =
  exp[γcv(P vy ,P v z )]∑
  z∈Ny exp[γc v(P vy ,P v z )]
  , (6)
  where γ is a temperature parameter controlling the smoothness of the normalization. In the semantic space, we have similar definitions of hs(·), cs(·, ·) and as(·, ·). In the following, we will use P vy [t] and P sy [t] to represent the visual and semantic prototype of class y at the t
  th step of propagation. We will use | · | to represent the size of a set and ReLU(·) , max{·, 0}. After τ steps of propagation, we concatenate the prototypes achieved in the two spaces for each class to form the final prototypes:
  Py = [P v y [τ ],P s y [τ ]], ∀ y ∈ YT . (7)
  The formulation of as(·, ·) and cs(·, ·) follows the same way as av(·, ·) and cv(·, ·). To be simple, we don’t rewrite the formulations here.
sections.heading: 4.4 CLASSIFIER GENERATED FROM PROTOTYPES
sections.text: Given the final prototypes and a query/test image x, we apply a fully-connected layer to predict the similarity between x and each class’s prototype. For each class y ∈ YT , it computes
  f(x,Py) = wReLU(W (1)x+W (2)Py + b (1)) + b, (8)
  Intuitively, the above classifier can be interpreted as a distance classifier with distance metric f(·, ·). When applied to a query image x, the predicted class is the most probable class, i.e., ŷ = argmaxy∈YT Pr(y|x;P ), where Pr denotes the probability.
sections.heading: 4.5 TRAINING STRATEGIES
sections.text: Episodic training on subgraphs. In each episode, we sample a subset of training classes YT ⊆ Yseen, and then build a training (or support) set DTtrain and a validation (or query) set DTvalid from the training samples of YT . They are two disjoint sets of samples from the same subset of training classes. We only use DTtrain to generate the prototypes P by the aforementioned procedures, and then apply backpropagation to minimize the loss of DTvalid. This equals to an empirical risk minimization, i.e.,
  min ∑
  YT∼Yseen ∑ (x,y)∼DTvalid − log Pr(y|x;P ), (9)
  This follows the same idea of meta-learning Ravi & Larochelle (2016): it aims to minimize the validation loss instead of the training loss and thus can further improve generalization and restrain overfitting. Moreover, since we train the IPN model on multiple random tasks T and optimize the propagation scheme over different subgraphs (each generated for a random subset of classes), the generalization capability of IPN to unseen classes and new tasks can be substantially enhanced by this meta-training strategy. Moreover, it effectively reduces the propagation costs since computing the attention needs O(|YT |2) operations. In addition, as shown in the experiments, episodic training is effective in mitigating the problem of data imbalance and overfitting to seen classes, which harasses many ZSL methods.
  Consistency loss for alignment of visual and semantic space. In ZSL, the interaction and alignment between different spaces can provide substantial information about class dependency and is critical to the generation of accurate class prototypes. In IPN, we apply prototype propagation in two separate spaces on their own category graphs by using two untied attention modules. In order to keep the propagation in the two spaces consistent, we add an extra consistency loss to the training objective in Eq. (9). It computes the KL-divergence between two distributions defined in the two spaces for each class y ∈ YT at every propagation step t:
  DKL(p v y[t]||psy[t]) = − ∑ z∈YT pvy[t]z · log psy[t]z pvy[t]z , (10)
  where
  pvy[t]z , exp[cv(P vy [t],P v z [t])]∑
  z∈YT exp[c v(P vy [t],P v z [t])]
  , (11)
  psy[t]z , exp[cs(P sy [t],P s z [t])]∑
  z∈YT exp[c s(P sy [t],P s z [t])]
  . (12)
  The consistency loss enforces the two attention modules in the two spaces to generate the same attention scores over classes, even when the prototypes and attention module’s parameter are different in the two spaces. In this way, they share similar class dependency over the propagation steps. So the generated dual prototypes are well aligned and can provide complementary information when generating the classifier. The final optimization we used to train IPN becomes:
  min ∑
  YT∼Yseen  ∑ (x,y)∼DTvalid − log Pr(y|x;P ) + λ τ∑ t=1 ∑ y∈YT DKL(p v y[t]||psy[t])  , (13) where λ is weight for the consistency loss, and P , pvy[t], and p s y[t] are computed only based on DTtrain, the training (or support) set of task-T for classes in YT .
sections.heading: 5 EXPERIMENTS
sections.text: We evaluate IPN and compare it with several state-of-the-art ZSL models in both ZSL setting (where the test set is only composed of unseen classes) and generalized ZSL setting (where the test set is composed of both seen and unseen classes). We report three standard ZSL evaluation metrics, and provide ablation study of several variants of IPN in the generalized ZSL setting. The visualization of the learned class representation is shown in Figure 1.
sections.heading: 5.1 DATASETS AND EVALUATION CRITERIA
sections.text: Small and Medium benchmarks. Our evaluation includes experiments on three standard benchmark datasets widely used in previous works, i.e., AWA2 (Xian et al., 2019a), CUB (Welinder et al., 2010) and aPY (Farhadi et al., 2009). The former two are extracted from ImageNet-1K, where AWA2 contains 50 animals classes and CUB contains 200 bird species. aPY includes 32 classes. Detailed statistics of them can be found in Table 1.
  Two new large-scale benchmarks. The three benchmark datasets above might be too small for modern DNN models since the training may suffers from large variance and end-to-end training usually fails. Previous ZSL methods therefore fine-tune a pre-trained CNN to achieve satisfactory performance on these datasets, which is costly and less practical. Given these issues, we proposed two new large-scale datasets extracted from tieredImageNet (Ren et al., 2018), i.e., tieredImageNetSegregated and tieredImageNet-Mixed, whose statistics are given in Table 1. Comparing to the three benchmarks, they are over 10× larger, and training a ResNet101 (He et al., 2016) from scratch on their training sets can achieve a validation accuracy of 65%, indicating that end-to-end training without fine-tuning is possible. These two benchmarks differ in that the “Mixed” version allows a seen class and an unseen class to belong to the same super category, while the “Segregated” version does not. The class attributes are the word embeddings of the class names provided by GloVe (Pennington et al., 2014). We removed the classes that do not have embeddings in GloVe.
  Following the most popular setting of ZSL (Xian et al., 2019a), we report the per-class accuracy and harmonic mean of the accuracy on seen and unseen classes.
sections.heading: 5.2 IMPLEMENTATION AND HYPERPARAMETER DETAILS
sections.text: For a fair comparison, we follow the setting in (Xian et al., 2019a) and use a pre-trained ResNet101 (He et al., 2016) to extract 2048-dimensional image features without fine-tuning. For the three standard benchmarks, the model is pre-trained on ImageNet-1K. For the two new datasets, it is pre-trained on the training set used to train ZSL models.
  All hyperparameters were chosen on the validation sets provided by Xian et al. (2019a). We use the same hyperparameters tuned on AWA2 for other datasets since they are of similar data type, except for aPY on which we choose learning rate 1.0× 10−3 and weight decay 5.0× 10−4. For the rest of the datasets, we train IPN by Adam (Kingma & Ba, 2015) for 360 epochs with a weight decay factor 1.0× 10−4. The learning rate starts from 2.0× 10−5 and decays by a multiplicative factor 0.1 every 240 epochs. In each epoch, we update IPN on multiple N -way-K-shot tasks (N -classes with K samples per class), each corresponding to an episode mentioned in Section 4.5. In our experiments, the number of episodes in each epoch is n/NK, where n is the total size of the training set, N = 30 and K = 1 (a small K can mitigate the imbalance between seen and unseen classes); hv and hs are linear transformations; we set temperature γ = 10, threshold = cos 40o for the cosine similarities between class prototypes, and weight for consistency loss λ = 1. We use propagation steps τ = 2.
sections.heading: 5.3 MAIN RESULTS
sections.text: The results are reported in Table 2. Although most previous works (Xian et al., 2019b;a; Sariyildiz & Cinbis, 2019) show that the more complicated generative models usually outperforms the discriminative ones. Our experiments imply that IPN, as a simple discriminative model, can significantly outperform the generative models requiring much more training efforts. Most baselines achieve a
  high ACCseen but much lower ACCunseen since they suffer from and more sensitive to the imbalance between seen and unseen classes. In contrast, IPN always achieves the highest H-metric among all methods, indicating less overfitting to seen classes. This is due to the more accurate prototypes achieved by iterative propagation in two interactive spaces across seen and unseen classes, and the episodic training strategy, which effectively improves the generalization.
  In scaling the experiments to the two large datasets, discriminative models perform better than generative models, which are easily prone to overfitting or divergence. Moreover, during the test stage, generative models require to synthesize data for unseen classes, which may lead to expensive computations when applied to tasks with many classes. In contrast, IPN scales well to these large datasets, outperforming state-of-the-art baselines by a large margin. The performance of most models on tieredImageNet-Segregated is generally worse than tieredImageNet-Mixed because that the unseen classes in the former do not share the same ancestor classes with the seen classes, while in the latter such sharing is allowed.
sections.heading: 5.4 ABLATION STUDY
sections.text: Are two prototypes necessary? We compare IPN with its two variants, each using only one out of the two types of prototypes from the two spaces when generating the classifier. The results are shown in Figure 3. It shows that semantic prototype (only) performs much better than visual prototype (only), especially on the unseen classes. This is because that the unseen classes’ semantic embedding is given while its visual information is missing (zero images) during the test stage. Although IPN is capable of generating a visual prototype based on other classes’ images and class dependency, it is much less accurate than the semantic prototype. However, it also shows that the visual prototypes (even the ones of unseen classes) can provide extra information and further improve the performance of the semantic prototype, e.g., they improve the unseen classes’ accuracy from 61.1% to 67.5%. One possible reason is that the visual prototype bridges the query image’s visual feature with the semantic features of classes, and thus provides an extra “visual context” for the localization of the correct class. In addition, we observe that IPN with only the visual prototype converges faster than IPN with only a semantic prototype (30 epochs vs. 200 epochs). This might be an artifact caused by the pre-trained CNN, which already produces stable visual prototypes since the early stages. However, it raises a caveat that heavily relying on a pre-trained model might discourage the learning and results in a bad local optimum.
  How’s the benefit of propagation? We then evaluate the effects of the propagation scheme in the two spaces. In particular, we compare IPN with its variants that apply the propagation steps in only one of the two spaces, and the variant that does not apply any propagation after initialization. Their performance is reported in Figure 3. Without any propagation, the classification performance heavily relies on pre-trained CNN. Hence, the accuracy of seen classes can be much higher than that of the unseen classes. Compared to IPN without any propagation, applying propagation in either the visual space or the semantic space can improve the accuracy of unseen classes by 2∼3%, while the accuracy on the seen classes stays almost the same. When equipped with dual propagation, we
  observe a significant improvement of the accuracy of the unseen classes but a degradation on the seen classes. However, they lead to the highest H-metric, which indicates that the dual propagation achieves a desirable trade-off between overfitting on seen classes and generalization to unseen classes.
  Effect of Isometric Alignment In IPN, we minimize a consistency loss during training for alignment of the visual and semantic space. Here, we consider two other options for alignment that exchange the attention modules used in the two spaces, i.e., using the visual attention module in the semantic space, or using the semantic attention module in the visual space. We also include the IPN trained without consistency loss or any alignment as a baseline. Their results are reported in Figure 3. Compared to IPN with/without consistency loss, sharing a single attention module between the two spaces suffers from the overfitting problem on the seen classes. This is because that the visual features and semantic embeddings are two different types of features so they should not share the same similarity metric. In contrast, the consistency loss only enforces the output similarity scores of the two attention modules to be close, while the models/functions computing the similarity can be different. Therefore, to produce the same similarity scores, it does not require each class having the same features in the two spaces. Hence, it leads to a much softer regularization.
sections.heading: 6 SENSITIVITY OF HYPERPARAMETERS
sections.text: The proposed IPN has some hyperparameters as mentioned in Section 5.2. To analyze the sensitivity of those hyperparameters, such as threshold, temperature, N and K for N-way-K-shot task, propagation steps, etc. We try different values and show the performance on the AWA2 dataset in Figure 4. The temperature γ is not sensitive for the accuracy. The threshold needs to be bigger than cosine(30o) to give IPN some space for the filtering. The propagation steps τ is best at 2. Weight decay is a sensitive hyperparameter and this is also found in previous ZSL literature (Zhang & Shi, 2019; Liu et al., 2019d).
sections.heading: 7 CONCLUSIONS
sections.text: In this work, we propose an Isometric Propagation Network, which learns to relate classes within each space and across two spaces. IPN achieves state-of-the-art performance and we show the importance of each component and visualize the interpretable relationship of the learned class representations. In future works, we will study other possibilities to initialize the visual prototypes of unseen classes in IPN.
sections.heading: A EVALUATION CRITERIA
sections.text: Considering the imbalance between test classes, by following the most recent works (Xian et al., 2019a), given a test set D, we evaluate ZSL methods’ performance based on the per-class accuracy averaged over a targeted set of classes Y , i.e.,
  ACCY = 1 |Y| ∑ y∈Y 1 |Dy| ∑
  (x,y)∈Dy
  1[ŷ = y], (14)
  Dy , {(xi, yi) ∈ D : yi = y}.
  Following Xian et al. (2019a), we report ACCY when Y = Yseen and Y = Yunseen, and the harmonic mean of them, i.e.,
  H = 2 ·ACCYseen ·ACCYunseen ACCYseen +ACCYunseen . (15)
sections.heading: B EXPERIMENTS ON ZERO-SHOT LEARNING SETTING
sections.text: The comparison between IPN to other baselines on the setting of Zero-shot Learning is shown in Table 3. On CUB, AWA2, aPU, tiered-M, and tiered-S, our IPN outperforms other algorithms by a large margin.
sections.heading: C MORE EXPERIMENTS ON LARGE-SCALE DATASETS
sections.text: In our work, we follow the train/test split suggested by Frome et al. (2013), who proposed to use the 21K ImageNet dataset for zero-shot evaluation. Here, 2-hops and 3-hops represent that the evaluation is over the classes which are 2/3 hops away from ImageNet-1k. Different from experiments on AWA2, we use N=20 and K=5 for the full ImageNet. Besides, since the edges between each class are provided on the full ImageNet, we directly use them instead of calculating by Eq. (3).
title: ISOMETRIC PROPAGATION NETWORK FOR GENERALIZED ZERO-SHOT LEARNING