Deep Multi-View Learning via Task-Optimal CCA
Canonical Correlation Analysis (CCA) is widely used for multimodal data analysis and, more recently, for discriminative tasks such as multi-view learning; however, it makes no use of class labels. Recent CCA methods have started to address this weakness but are limited in that they do not simultaneously optimize the CCA projection for discrimination and the CCA projection itself, or they are linear only. We address these deficiencies by simultaneously optimizing a CCA-based and a task objective in an end-to-end manner. Together, these two objectives learn a non-linear CCA projection to a shared latent space that is highly correlated and discriminative. Our method shows a significant improvement over previous state-of-the-art (including deep supervised approaches) for cross-view classification (8.5% increase), regularization with a second view during training when only one view is available at test time (2.2-3.2%), and semi-supervised learning (15%) on real data.
This paper addresses the problem of jointly performing CCA with task labeling. The problem is timely and important as it is challenging to perform CCA jointly with the task classification (see below) and hence previous work typically perform this in a pipeline - that is, first projecting the data using a pre-trained CCA and then training a task classifier using the projected representation. As the authors note, this may be problematic as CCA may delete important information that is relevant for the classification, if training is not done jointly. As the authors note, the main challenge in developing a task-optimal form of deep CCA that discriminates based on the CCA projection is in computing this projection within the network. To deal with this, the authors propose approximations for two steps: maximizing the sum correlation between activations A1 and A2 and enforcing orthonormality constraints within A1 and A2. Particularly, the authors present three methods that progressively relax the orthogonality constraints. Also, because correlation is not separable across batches, SGD is not possible for deep CCA training and the authors deal with this too (although this was dealt with in previous work on deep CCA as well). This is an empirical paper in the sense that no guarantees are provided for the proposed techniques. The experiments are thorough, convincing and the span a range of applications. The results demonstrate the value of the proposed approximations, and I hence recommend a weak accept of the paper.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Deep Multi-View Learning via Task-Optimal CCA
Canonical Correlation Analysis (CCA) is widely used for multimodal data analysis and, more recently, for discriminative tasks such as multi-view learning; however, it makes no use of class labels. Recent CCA methods have started to address this weakness but are limited in that they do not simultaneously optimize the CCA projection for discrimination and the CCA projection itself, or they are linear only. We address these deficiencies by simultaneously optimizing a CCA-based and a task objective in an end-to-end manner. Together, these two objectives learn a non-linear CCA projection to a shared latent space that is highly correlated and discriminative. Our method shows a significant improvement over previous state-of-the-art (including deep supervised approaches) for cross-view classification (8.5% increase), regularization with a second view during training when only one view is available at test time (2.2-3.2%), and semi-supervised learning (15%) on real data.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper addresses the problem of jointly performing CCA with task labeling. The problem is timely and important as it is challenging to perform CCA jointly with the task classification (see below) and hence previous work typically perform this in a pipeline - that is, first projecting the data using a pre-trained CCA and then training a task classifier using the projected representation. As the authors note, this may be problematic as CCA may delete important information that is relevant for the classification, if training is not done jointly. As the authors note, the main challenge in developing a task-optimal form of deep CCA that discriminates based on the CCA projection is in computing this projection within the network. To deal with this, the authors propose approximations for two steps: maximizing the sum correlation between activations A1 and A2 and enforcing orthonormality constraints within A1 and A2. Particularly, the authors present three methods that progressively relax the orthogonality constraints. Also, because correlation is not separable across batches, SGD is not possible for deep CCA training and the authors deal with this too (although this was dealt with in previous work on deep CCA as well). This is an empirical paper in the sense that no guarantees are provided for the proposed techniques. The experiments are thorough, convincing and the span a range of applications. The results demonstrate the value of the proposed approximations, and I hence recommend a weak accept of the paper.
assistant
This paper addresses the problem of jointly performing CCA with task labeling. The problem is timely and important as it is challenging to perform CCA jointly with the task classification (see below) and hence previous work typically perform this in a pipeline - that is, first projecting the data using a pre-trained CCA and then training a task classifier using the projected representation. As the authors note, this may be problematic as CCA may delete important information that is relevant for the classification, if training is not done jointly. As the authors note, the main challenge in developing a task-optimal form of deep CCA that discriminates based on the CCA projection is in computing this projection within the network. To deal with this, the authors propose approximations for two steps: maximizing the sum correlation between activations A1 and A2 and enforcing orthonormality constraints within A1 and A2. Particularly, the authors present three methods that progressively relax the orthogonality constraints. Also, because correlation is not separable across batches, SGD is not possible for deep CCA training and the authors deal with this too (although this was dealt with in previous work on deep CCA as well). This is an empirical paper in the sense that no guarantees are provided for the proposed techniques. The experiments are thorough, convincing and the span a range of applications. The results demonstrate the value of the proposed approximations, and I hence recommend a weak accept of the paper.