Domain-Invariant Projection Learning for Zero-Shot Recognition
Zero-shot learning (ZSL) aims to recognize unseen object classes without any training samples, which can be regarded as a form of transfer learning from seen classes to unseen ones. This is made possible by learning a projection between a feature space and a semantic space (e.g. attribute space). Key to ZSL is thus to learn a projection function that is robust against the often large domain gap between the seen and unseen classes. In this paper, we propose a novel ZSL model termed domain-invariant projection learning (DIPL). Our model has two novel components: (1) A domain-invariant feature self-reconstruction task is introduced to the seen/unseen class data, resulting in a simple linear formulation that casts ZSL into a min-min optimization problem. Solving the problem is non-trivial, and a novel iterative algorithm is formulated as the solver, with rigorous theoretic algorithm analysis provided. (2) To further align the two domains via the learned projection, shared semantic structure among seen and unseen classes is explored via forming superclasses in the semantic space. Extensive experiments show that our model outperforms the state-of-the-art alternatives by significant margins.
The authors present an algorithm for zero-shot learning based on learning a linear projection between the features of a pre-trained convnet and a semantic space in which to do nearest neighbors classification.  Overall, I found the exposition of the paper very confusing, and I am still struggling to understand the exact set-up in the experiments and theorems after several re-readings. I'm not 100% certain from the paper what the authors are actually doing and what they have available as training data. In particular, the authors denote the test images by D_u, which they are sure to point out are unlabeled. They then define l_i^{(u)} to be the label of test point x_i^{(u)} - how can the label be included in a set of test images that is unlabeled? What information from D_u is provided when learning the projection matrix W, and what is held out until test time? What is the algorithm used to map the projection of a feature to a class? I *think* what is happening is that the semantic features y_j for each unseen class j are provided at training time, but not the label for which class each test image belongs to, but this is not well-explained. It is also not clear to me how these vectors y_j are chosen in the experiments. I *think* that the datasets they work with have pre-defined attribute vectors for every class - providing a concrete example of such an attribute vector would help explain the source of these y_j terms.  In the loss function in Eq 1, I am not clear on why W and W^T are used for the forward/backward projection from visual features to semantic features, rather than W and its Moore-Penrose pseudoinverse. Is the learned W approximately orthogonal?  Line 152: "contentional" is not a word. I am not sure what the authors are trying to say here.  Section 3.3: This section is especially confusing. I suppose the confusion stems from the fact that f_i^{(t)} is a vector, and you are doing an optimization over elements of the vector, that is, just taking the minimum element. Not to mention some of the math seems wrong - you define the gradient of the minimum of the element of the vector to be 1/number of elements with that value. The minimum is a piecewise-linear function, and if multiple elements of the vector have the same (minimum) value, the minimum does not have a well-defined gradient at that point and instead has a *sub*gradient, which is a set of directions that fall below the function.  Section 3.4: This section is far too dense, to the point of being almost unreadable. I would recommend reformatting this and, if space is not available, just state the main result of the proof and move the body to the supplementary materials. Also I am not sure that Proposition 1 is novel enough to warrant a proof here - presumably the theory of convergence for Sylvester equations is well-studied and the appropriate theorem could simply be cited here.  I appreciate the inclusion of ablation studies. It should be more common in NIPS submissions.  Overall, while I'm somewhat skeptical of how realistic the transductive ZSL setting is (since information about unseen classes is not likely to be available at training time) and I am not certain that the derivation of the updates is correct, the empirical performance does seem very good, and the results are a significant improvement on the aPY dataset.