Deep neural networks have been tremendously successful in a number of tasks. One of the main reasons for this is their capability to automatically learn representations of data in levels of abstraction, increasingly disentangling the data as the internal transformations are applied. In this paper we propose a novel regularization method that penalize covariance between dimensions of the hidden layers in a network, something that benefits the disentanglement. This makes the network learn nonlinear representations that are linearly uncorrelated, yet allows the model to obtain good results on a number of tasks, as demonstrated by our experimental evaluation. The proposed technique can be used to find the dimensionality of the underlying data, because it effectively disables dimensions that aren't needed. Our approach is simple and computationally cheap, as it can be applied as a regularizer to any gradient-based learning model.
This paper presents a regularization mechanism which penalizes covariance between all dimensions in the latent representation of a neural network. This penalty is meant to disentangle the latent representation by removing shared covariance between each dimension. 
While the proposed penalty is described as a novel contribution, there are multiple instances of previous work which use the same type of penalty (Cheung et. al. 2014, Cogswell et. al. 2016). Like this work, Cheung et. al. 2014 propose the XCov penalty which penalizes cross-covariance to disentangle subsets of dimensions in the latent representation of autoencoder models. Cogswell et. al. 2016 also proposes a similar penalty (DeCov) to this work for reducing overfitting in supervised learning.
The novel contribution of the regularizer proposed in this work is that it also penalizes the variance of individual dimensions along with the cross-covariance. Intuitively, this should lead to dimensionality reduction as the model will discard variance in dimensions which are unnecessary for reconstruction. But given the similarity to previous work, the authors need to quantitatively evaluate the value in additionally penalizing variance of each dimension as compared with earlier work. Cogswell et. al. 2016 explicitly remove these terms from their regularizer to prevent the dynamic range of the activations from being unnecessarily rescaled.  It would be helpful to understand how this approach avoids this issues - i.e.,  if you penalize all the variance terms then you could just be arbitrarily rescaling the activities, so what prevents this trivial solution?
There doesn't appear to be a definition of the L1 penalty this paper compares against and it's unclear why this is a reasonable baseline. The evaluation metrics this work uses (MAPC, CVR, TdV, UD) need to be justified more in the absence of their use in previous work. While they evaluate their method on non-toy dataset such as CIFAR, they do not show what the actual utility of their proposed regularizer serves for such a dataset beyond having no-regularization at all. Again, the utility of the evaluation metrics proposed in this work is unclear.
The toy examples are kind of interesting but it would be more compelling if the dimensionality reduction aspect extended to real datasets.
> Our method has no penalty on the performance on tasks evaluated in the experiments, while it does disentangle the data
This needs to be expanded in the results as all the results presented appear to show Mean Squared Error increasing when increasing the weight of the regularization penalty.