Learning a better representation with neural networks is a challenging problem, which has been tackled from different perspectives in the past few years. In this work, we focus on learning a representation that would be useful in a clustering task. We introduce two novel loss components that substantially improve the quality of produced clusters, are simple to apply to arbitrary models and cost functions, and do not require a complicated training procedure. We perform an extensive set of experiments, supervised and unsupervised, and evaluate the proposed loss components on two most common types of models, Recurrent Neural Networks and Convolutional Neural Networks, showing that the approach we propose consistently improves the quality of KMeans clustering in terms of mutual information scores and outperforms previously proposed methods.
Summary
This paper proposes two regularizers that are intended to make the
representations learned in the penultimate layer of a classifier more conforming
to inherent structure in the data, rather than just the class structure enforced
by the classifier. One regularizer encourages the weights feeding into the
penultimate layer to be dissimilar and the other encourages the activations
across samples (even if they belong to the same class) to be dissimilar.
Pros
- The proposed regularizers are able to separate out the classes inherent in the
  data, even if this information is not provided through class labels. This is
validated on several datasets using visualizations as well as quantitative
metrics based on mutual information.
Cons
- It is not explained why it makes sense to first convert the weight vectors
  into probability distributions by applying the softmax function, and then
measuring distances using KL divergence between the probability distributions.
It should be explained more clearly if there is there a natural interpretation
of the weight vectors as probability distributions. Otherwise it is not obvious
why the distance between the weight vectors is measured the way it is.
- Similarly, the ReLU activations are also first converted into probability
  distributions by applying a softmax. It should be explained why the model does
this, as opposed to simply using dot products to measure similarity.
- The model is not compared to simpler alternatives such as adding an
  orthogonality regularization on the weights, i.e., computing W^TW and making
the diagonals close to 1 and all other terms 0. Similar regularizers can be
applied for activation vectors as well.
- The objective of this paper seems to be to produce representations that are
  easy to separate into clusters. This topic has a wealth of previous work. Of
particular relevance are methods such as t-SNE [1], parametric t-SNE [2], and
DEC [3]. The losses introduced in this paper are fairly straight-forward.
Therefore it would be good to compare to these baselines to show that a simple
loss function is sufficient to achieve the objective.
- Disentangling usually refers to disentangling factors of variation, for
  example, lighting, pose, and object identity which affect the appearance of a
data point. This is different from separability, which is the property of a
representation that makes the presence of clusters evident. This paper seems to
be about learning separable representations, whereas the title suggests that it
is about disentangled ones. 
Quality
The design choices made in the paper (such as the choice of distance function)
is not well explained. Also, given that the modifications introduced are quite
simple, it can be improved by doing more thorough comparisons to other
baselines.
Clarity
The paper is easy to follow.
Originality
The novel aspect of the paper is the way distance is measured by converting the
weights (and activations) to probability distributions and using KL divergence
to measure distance. However, it is not explained what motivated this choice.
Significance
The objective of this model is to produce representations that are separable, which
is of general interest. However, given the wealth of previous work done in
clustering, this paper would only be impactful if it compares to other hard
baselines and shows clear advantages.
[1] van der Maaten, Laurens and Hinton, Geoffrey. Visualizing
data using t-SNE. JMLR, 2008.
[2] van der Maaten, Laurens. Learning a parametric embedding
by preserving local structure. In International Conference
on Artificial Intelligence and Statistics, 2009.
[3] Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for
clustering analysis. ICML 2016.