Disentangled activations in deep networks
Deep neural networks have been tremendously successful in a number of tasks. One of the main reasons for this is their capability to automatically learn representations of data in levels of abstraction, increasingly disentangling the data as the internal transformations are applied. In this paper we propose a novel regularization method that penalize covariance between dimensions of the hidden layers in a network, something that benefits the disentanglement. This makes the network learn nonlinear representations that are linearly uncorrelated, yet allows the model to obtain good results on a number of tasks, as demonstrated by our experimental evaluation. The proposed technique can be used to find the dimensionality of the underlying data, because it effectively disables dimensions that aren*t needed. Our approach is simple and computationally cheap, as it can be applied as a regularizer to any gradient-based learning model.
 The authors propose a penalization term that enforces decorrelation between the dimensions of the representation They show that it can be included as additional term in cost functions to train generic models. The idea is simple and it seems to work for the presented examples. However, they talk about gradient descent using this extra term, but I*d like to see the derivatives of the proposed term depending on the parameters of the model (and this depends on the model!). On the other hand, given the expression of the proposed regulatization, it seems to lead to non-convex optimization problems which are hard to solve. Any comment on that?. Moreover, its results are not quantitatively compared to other Non-Linear generalizations of PCA/ICA designed for similar goals (e.g. those cited in the "related work" section or others which have been proved to be consistent non-linear generalizations of PCA such as: Principal Polynomial Analysis, Dimensionality Reduction via Regression that follow the family introduced in the book of Jolliffe, Principal Component Analysis). Minor points: Fig.1 conveys not that much information.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Disentangled activations in deep networks
Deep neural networks have been tremendously successful in a number of tasks. One of the main reasons for this is their capability to automatically learn representations of data in levels of abstraction, increasingly disentangling the data as the internal transformations are applied. In this paper we propose a novel regularization method that penalize covariance between dimensions of the hidden layers in a network, something that benefits the disentanglement. This makes the network learn nonlinear representations that are linearly uncorrelated, yet allows the model to obtain good results on a number of tasks, as demonstrated by our experimental evaluation. The proposed technique can be used to find the dimensionality of the underlying data, because it effectively disables dimensions that aren*t needed. Our approach is simple and computationally cheap, as it can be applied as a regularizer to any gradient-based learning model.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The authors propose a penalization term that enforces decorrelation between the dimensions of the representation They show that it can be included as additional term in cost functions to train generic models. The idea is simple and it seems to work for the presented examples. However, they talk about gradient descent using this extra term, but I*d like to see the derivatives of the proposed term depending on the parameters of the model (and this depends on the model!). On the other hand, given the expression of the proposed regulatization, it seems to lead to non-convex optimization problems which are hard to solve. Any comment on that?. Moreover, its results are not quantitatively compared to other Non-Linear generalizations of PCA/ICA designed for similar goals (e.g. those cited in the "related work" section or others which have been proved to be consistent non-linear generalizations of PCA such as: Principal Polynomial Analysis, Dimensionality Reduction via Regression that follow the family introduced in the book of Jolliffe, Principal Component Analysis). Minor points: Fig.1 conveys not that much information.
assistant
 The authors propose a penalization term that enforces decorrelation between the dimensions of the representation They show that it can be included as additional term in cost functions to train generic models. The idea is simple and it seems to work for the presented examples. However, they talk about gradient descent using this extra term, but I*d like to see the derivatives of the proposed term depending on the parameters of the model (and this depends on the model!). On the other hand, given the expression of the proposed regulatization, it seems to lead to non-convex optimization problems which are hard to solve. Any comment on that?. Moreover, its results are not quantitatively compared to other Non-Linear generalizations of PCA/ICA designed for similar goals (e.g. those cited in the "related work" section or others which have been proved to be consistent non-linear generalizations of PCA such as: Principal Polynomial Analysis, Dimensionality Reduction via Regression that follow the family introduced in the book of Jolliffe, Principal Component Analysis). Minor points: Fig.1 conveys not that much information.