Deep neural networks (DNNs) typically have enough capacity to fit random data by brute force even when conventional data-dependent regularizations focusing on the geometry of the features are imposed. We find out that the reason for this is the inconsistency between the enforced geometry and the standard softmax cross entropy loss. To resolve this, we propose a new framework for data-dependent DNN regularization, the Geometrically-Regularized-Self-Validating neural Networks (GRSVNet). During training, the geometry enforced on one batch of features is simultaneously validated on a separate batch using a validation loss consistent with the geometry. We study a particular case of GRSVNet, the Orthogonal-Low-rank Embedding (OLE)-GRSVNet, which is capable of producing highly discriminative features residing in orthogonal low-rank subspaces. Numerical experiments show that OLE-GRSVNet outperforms DNNs with conventional regularization when trained on real data. More importantly, unlike conventional DNNs, OLE-GRSVNet refuses to memorize random data or random labels, suggesting it only learns intrinsic patterns by reducing the memorizing capacity of the baseline DNN.
The paper proposes a data-dependent regularization method which is coupled with softmax loss to train deep neural networks for classification. The paper turns to Orthogonal Low-rank Embedding (OLE) loss for the geometric constraint that one class of data/feature are assumed to reside on a low-rank subspace that subspaces of different classes are orthogonal ideally. The probability in the softmax is then modeled as cosine similarity between data feature and the class-specific subspaces. In this way, geometric loss and softmax loss have the common goal for optimization. Moreover, during training, the geometry enforced on one batch of features is simultaneously validated on a separate batch using a validation loss. The experiments seem to suggest such a model helps avoid overfitting/memorizing noisy training data. The paper reads well and is easy to follow.
However, the paper is limited in technical novelty and practical significance. Here are some concerns -- 
1) The paper only studies one method based on OLE, though it cites the center loss [19]. How does the center loss behave in face of noisy training label? Would it also be able to refuse to fit the noisy training data?
2) As each class has its own (low-rank) subspace, and the rank is reduced by imposing the nuclear norm. It seems that the proposed method is hard to extend to many classes (class number is larger than the dimension)?
3) The datasets in the experiments are quite small in scale and class number. It is not persuasive unless tested on larger scale data or with large class number.
4) The proposed method seems to be limited in deal with discrete labels (e.g., classification), is it easy to extend to continuous target, say regression problems like depth estimation and surface normal estimation?
5) While the authors claim as a main contribution that the proposed GRSVNet is a general framework, it is hard to see how this framework can be used in other tasks other than classification.
6) The experiments are less persuasive. It's better to add the error bar to see the improvement by the proposed method is not due to random initialization. Running time should also be compared, as nuclear norm seems to be time consuming.