This paper proposes a neural network for classification and regression, without the need to learn layout structures in the output space. Standard solutions such as softmax cross-entropy and mean squared error are effective but parametric, meaning that known inductive structures such as maximum margin separation and simplicity (Occam's Razor) need to be learned for the task at hand. Instead, we propose polar prototype networks, a class of networks that explicitly states the structure, i.e., the layout, of the output. The structure is defined by polar prototypes, points on the hypersphere of the output space. For classification, each class is described by a single polar prototype and they are a priori distributed with maximal separation and equal shares on the hypersphere. Classes are assigned to prototypes randomly or based on semantic priors and training becomes a matter of minimizing angular distances between examples and their class prototypes. For regression, we show that training can be performed as a polar interpolation between two prototypes, arriving at a regression with higher-dimensional outputs. From empirical analysis, we find that polar prototype networks benefit from large margin separation and semantic class structure, while only requiring a minimal amount of output dimensions. While the structure is simple, the performance is on par with (classification) or better than (regression) standard network methods. Moreover, we show that we gain the ability to perform regression and classification jointly in the same space, which is disentangled and interpretable by design.
This paper unifies both classification and regression task based on the polar prototype network. For classification, the prototypes for all classes are chosen in advance based on a max-margin principle, while the embedding of all instances is then optimized to have small cosine distance to assigned prototypes. For the regression, the output value is interpolated between the two prototypes. Experiments on classification, regression, and combined tasks show the method can achieve good results.
The idea of using the prototype and the polar system is interesting, and the whole paper is well-written. However, there are still some problems and questions about this paper.
1. There are two problems with using the max-margin prototypes. First, to maximize the smallest distance between two prototypes, the authors use MC or evolutionary algorithms to do the optimization, which may be time-consuming, and it may be extremely difficult when the prototype space is high dimension. Second, the previous approach indeed obtains discriminative prototypes, but we lose the **class correlation**. In the extreme case, it is equal distance between all prototypes, but some similar classes will have a smaller prototype distance than others. For example, the prototype distance between "cat" and "dog" should not be the same as that between "car1" and "car2". The semantic consideration in the paper can solve this problem to some extent, but there needs more evidence.
Using the pre-defined prototype is also considered in the paper "M. Perrot et al. Regressive Virtual Metric Learning. NIPS15". 
2. For the unified output space
One main contribution is that based on the polar system, the method unifies both classification and regression tasks in the same space. We can also do this in basic embedding algorithms. In the embedding space, a method can do both classification and regression with the nearest neighbor rule (based on majority voting and average respectively). The authors should compare with such kinds of methods in the experiments.
3. Experiments
From the experiments, using semantic cannot improve a lot for the classification task. The authors can try more datasets to validate is this the common scenario. The reviewer strongly suggests the authors should compare with more methods. For example, in some papers the prototypes are learned simultaneously (Snell et al. Prototypical networks for few-shot learning. NIPS17; Wen et al. A discriminative feature learning approach
for deep face recognition. ECCV16); while in other cases, there are no prototypes as we optimize the triplet/contrastive loss directly. First, the authors can compare classification performance with these approaches; besides, some visualization results can also show the used prototypes or embeddings. 
The main advantage of the method is not stressed clearly in the experiments part. The authors can clarify it in later versions.
The final rating depends on the authors' response.