This paper concerns open-world classification, where the classifier not only needs to classify test examples into seen classes that have appeared in training but also reject examples from unseen or novel classes that have not appeared in training. Specifically, this paper focuses on discovering the hidden unseen classes of the rejected examples. Clearly, without prior knowledge this is difficult. However, we do have the data from the seen training classes, which can tell us what kind of similarity/difference is expected for examples from the same class or from different classes. It is reasonable to assume that this knowledge can be transferred to the rejected examples and used to discover the hidden unseen classes in them. This paper aims to solve this problem. It first proposes a joint open classification model with a sub-model for classifying whether a pair of examples belongs to the same or different classes. This sub-model can serve as a distance function for clustering to discover the hidden classes of the rejected examples. Experimental results show that the proposed model is highly promising.
The main goal of this paper is to cluster images from classes unseen during training.
This is an interesting extension of the open-world paradigm, where at test time, the classifier has to identify images beloning to the C seen classes during training, but also identify (reject) images which were previously unseen. These rejected images could be clustered to identify the number of unseen classes; either for revealing the underlying structure of the unseen classes, or to reduce annotation costs.
In order to do so, an extensive framework is proposed, consisting of 3 ConvNet architectures, followed by a hierarchical clustering approach. The 3 ConvNets all have a different goal:
1. an Open Classification Network (per class sigmoid, trained 1vsRest, with thresholds for rejection)
2. Pairwise Classification Network, (binary sigmoid, trained on pairs of images of same/different classes)
3. Auto encoder network
These network are jointly trained, and the joint-loss is simply the addition of a cross-entropy loss (from OCN), the binary cross-entropy loss (from PCN) and a pixel wise loss (from AE). 
Remarks:
- it is unclear if the ConvNet weights of the first layers are shared). 
- it is unclear how joint training might help, given that the objectives do not influence each other
- Eq 1: 
  *label "y_i" has two different semantics (L_ocn it is the class label, while in L_pcn it is the label of an image pair being from the same class or not)
  * s_j is undefined
  * relation between the p(y_i = 1) (in PCN) and g(x_p,x_q) in Eq 2 could be made more explicit, PCN depends on two images, according to Eq 1, it seems just a sum over single images.
- It is unclear why the Auto Encoder network is added, and what its function is.
- It is unclear wether OCN requires/uses unseen class examples during training.
- Last paragraph of 3.1 "The 1-vs-rest ... rejected", I don't see why you need 1vsRest classifiers for this, a multi-class (softmax) output can also be thresholded to reject an test image from the known classes and to assign it to the unknown class.
Experimental evaluation
The experimental evaluation uses 2 datasets, MNIST and EMNIST, both are very specific for character recognition. It is a pity that not also more general image classification has been considered (CIFAR100, ImageNet, Places365, etc), that would provide insights to the more general behaviour of the proposed ideas.
My major concern is that the clustering task is not extensively explored. Just a single setting (with a single random sampling of seen/unseen classes) has been evaluated. This is -in part- due to the nature of the chosen datasets, in a 10 class dataset it is difficult to show the influence of the number of unseen classes. So, I'd really urge the authors to extend this evaluation. Will the method discover more classes when 100 unknown classes are used? What kind of clusters are discovered? Are the types of classes in the seen/unseen classes important, I'd expect at least multiple runs of the current experiments on (E)MNIST. 
Further, I miss some baselines and ablation study. Questions which I'd like to seen answered: how good is the OCN representation when used for clustering compared to the PCN representation? What is the benefit of joint-training? How important is the AE in the loss?
Remaining remarks
- Just a very simple / non-standard ConvNet architecture is trained. Will a ResNet(32) show similar performance?
- In Eq 4, |C_i || y_j| seems a strange notation for union.
Conclusion
This paper brings in an interesting idea, is it possible to cluster the unseen classes in an open-world classification scenario?  A solution using a pairwise convnet followed by hierarchical clustering is proposed. This is a plausible solution, yet in total I miss an exploration of the solution. 
Both in terms of general visual classification (only MNIST is used, while it would be nice to see results on CIFAR and/or ImageNet as in Bendale&Boult 2016), as in exploration of different scenarios (different number of unseen classes, different samplings) and ablation of the method (independent training, using OCN for hierarchical clustering, influence of Auto Encoder). Therefore, I rate this paper as a (weak) reject: it is just not (yet) good enough for acceptance.