Many of the zero-shot learning methods have realized predicting labels of unseen images by learning the relations between images and pre-defined class-attributes. However, recent studies show that, under the more realistic generalized zero-shot learning (GZSL) scenarios, these approaches severely suffer from the issue of biased prediction, i.e., their classifier tends to predict all the examples from both seen and unseen classes as one of the seen classes. The cause of this problem is that they cannot properly learn a mapping to the representation space generalized to the unseen classes since the training set does not include any unseen class information. To solve this, we propose a concept to learn a mapping that embeds both images and attributes to the shared representation space that can be generalized even for unseen classes by interpolating from the information of seen classes, which we refer to shared manifold learning. Furthermore, we propose modality invariant variational autoencoders, which can perform shared manifold learning by training variational autoencoders with both images and attributes as inputs. The empirical validation of well-known datasets in GZSL shows that our method achieves the significantly superior performances to the existing relationbased studies.
The paper proposes an approach to generalized zero-shot learning by learning a shared latent space between the images and associated class-level attributes. To learn such a shared latent space and mapping for the same which is generalized and robust -- the authors propose ‘modality invariant variational autoencoders’ -- which allows one to perform shared manifold learning by training VAEs with both images and attributes as inputs. Empirical results demonstrate improvements over existing approaches on the harmonic mean metric present in the generalized zero-shot learning benchmark. Other than the concerns mentioned below, I like the basic idea adopted in the paper to extend Vedantam et. al. (2018)’s joint-VAEs (supporting unimodal inference) to the framework of generalized zero-shot learning. The proposed approach clearly results in improvements over baselines and existing approaches.
Comments:
- A minor correction. The paper claims the bias towards seen classes at inference for the existing GZSL approaches is due to the inability of obtaining training data for the unseen classes. In my opinion, this should be rephrased as the inability to learn a generalized enough representation (joint or otherwise) that is aware of the shift in distribution from seen to unseen classes (images or attributes) as this information is not available apriori.
- Writing Clarity Issues. In general, there is significant repetitions along certain lines throughout the introduction and approach. While the paper overall does a good job of explaining the motivation as well as the approach, some of the sections (and sentences within) could be written better to express the point being made. Specifically, the first paragraph in the introduction seems to be structured more from a few-shot setting. The paper would benefit from talking about few-shot learning first and then extending to the extreme setting of zero-shot learning. Similarly, the second paragraph in the introduction could be written more succinctly to express the point being made. The sentences -- “Moreover, it is difficult….widely available” -- are difficult to understand. Tables 4 and 5 should be positioned after the references section.
- A point repeatedly made in the paper suggests that learning unidirectional mappings from images to attributes (or otherwise) suffers from generalization to unseen classes. While I agree with this statement, most methods in GZSL hold out a subset of seen classes as validation (unseen) classes while learning such a mapping -- which I believe was also being done while learning the joint model in MIVAE (Can the authors confirm this? Is yes, how were these classes chosen?). As such, the authors should stress on the advantages learning a joint latent model over both modalities offers as opposed to unidirectional mappings while mentioning the above points.
- Learning the joint latent space for images and attributes has been referred to as learning a shared manifold in the paper -- with associated terms such as manifold representation being used as well. Sharing a latent space need not imply learning an entire manifold as the subspace captured by the latent space might as well be localized in the manifold in which it exists. Can the authors comment more on this connection with respect to the points around “shared manifold learning”?
- During inference, the authors operate in the latent space to find the most-relevant class by enumerating over all classes the KL-divergence between the unimodal encoder embeddings. Is there a particular reason the authors chose to operate in the latent space as opposed to operating in a modality space? Specifically, given an image the authors could have used the p(a|z) decoder to infer the attribute given the encoded z -- and subsequently finding the 1-nearest neighbor in that space. Any reason why this approach was not adopted? 
- On page 4, regarding the term L_dist in the objective for MIVAE, the authors draw the connections made in the appendix of Vedantam et. al. (2018) regarding the minimization of KL-divergence between the bimodal and a unimodal variational posterior(s). While the connection being made is accurate, the subsequent solution modes identified in the following paragraph -- “When equation 2 becomes minimum…” -- do not seem accurate. At minimality, unimodal encoders should be equivalent to the bimodal encoder marginalized over the absent rv under the conditional distribution of the data. Could the authors comment on whether the version presented in the paper is intended or is merely a typographical mistake?
Section 5 experiments suggest the learning rate used in practice was 10^3. Assuming a typo, this should be presumably 10^-3.
Experimental Issues. 
- The authors should explicitly mention if they are using the proposed split throughout all baselines and approaches for GZSL evaluations. It’s not explicitly mentioned in the text and is an important detail that should not be left out. Only the appendix mentions the number of seen/unseen classes.
- How did the authors select a validation split (held out seen classes) to train MIVAE? Did they directly borrow the training and validation splits present in the proposed split? Or did they create a split of their own? If latter, how was the split created? In general, I am curious about how the MIVAE checkpoint for inference was chosen.
- In section 5.2, the reasons in the 3rd paragraph elaborating \lambda_map=1 vs 0 not being too different for AWA and aPY are not clear. Could the authors comment a bit more on them?
The authors adressed the issues raised/comments made in the review. In light of my comments below to the author responses -- I am not inclined towards increasing my rating and will stick to my original rating for the paper.