Unseen Action Recognition with Unpaired Adversarial Multimodal Learning
In this paper, we present a method to learn a joint multimodal representation space that allows for the recognition of unseen activities in videos. We compare the effect of placing various constraints on the embedding space using paired text and video data. Additionally, we propose a method to improve the joint embedding space using an adversarial formulation with unpaired text and video data. In addition to testing on publicly available datasets, we introduce a new, large-scale text/video dataset. We experimentally confirm that learning such shared embedding space benefits three difficult tasks (i) zero-shot activity classification, (ii) unsupervised activity discovery, and (iii) unseen activity captioning.
 Summary: The paper aims to learn a common embedding space for video appearance and text caption features. The learned shared embedding space then allows multiple applications of zero-shot activity classification, unsupervised activity discovery and unseen activity captioning. The method is based on two autoencoders which have a common intermediate space. The losses optimized encourage the standard unimodal reconstructions in the AEs, along with joint embedding distances (appearance and text of the same video are encoded close by) as well as cross domain mapping (video encoding generates text and vice-versa), and cycle consistency. Apart from these additional supervised losses, unsupervised losses are added with adversarial learning which aim to bring the video and text encoding distributions in the common space close, as well as the standard real and generated video/text distributions close by adding corresponding discriminators (like in GANs). The whole system is learned end-to-end in two phases, first with supervised paired data and then with all paired and unpaired data. The experiments are shown on four datasets: ActivityNet, HMDB, UCF101, MLB-YouTube Positives: - The problem of multimodal learning is an interesting and challenging problem - The paper is novel; while the idea of a joint shared embedding space is not new this paper adds new losses as summarized above and shows reasonably convincing empirical results - The results are shown for diverse applications which highlight the generality of the method - The use of unpaired/unsupervised data is also relatively less explored which this paper incorporates as well - The empirical results given are convincing, eg Tab1 gives a good ablation study showing how the different components affect the performance. SoA comparison are given on a standard task (however see below) Negatives: - Comparison with state of the art result Tab2 should also contain the features used. The performances might just be higher due to the better features used (Kinetics pretrained I3D). Please give a self implemented baseline method with same features but some standard loss in the shared space to give an idea of the strength of the features. - Ideally features provided by previous papers’ authors should be used if available and it should be shown that the proposed method improves results. Overall the paper is well written and had novel aspects which are convincingly evaluated on challenging and diverse tasks.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Unseen Action Recognition with Unpaired Adversarial Multimodal Learning
In this paper, we present a method to learn a joint multimodal representation space that allows for the recognition of unseen activities in videos. We compare the effect of placing various constraints on the embedding space using paired text and video data. Additionally, we propose a method to improve the joint embedding space using an adversarial formulation with unpaired text and video data. In addition to testing on publicly available datasets, we introduce a new, large-scale text/video dataset. We experimentally confirm that learning such shared embedding space benefits three difficult tasks (i) zero-shot activity classification, (ii) unsupervised activity discovery, and (iii) unseen activity captioning.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 Summary: The paper aims to learn a common embedding space for video appearance and text caption features. The learned shared embedding space then allows multiple applications of zero-shot activity classification, unsupervised activity discovery and unseen activity captioning. The method is based on two autoencoders which have a common intermediate space. The losses optimized encourage the standard unimodal reconstructions in the AEs, along with joint embedding distances (appearance and text of the same video are encoded close by) as well as cross domain mapping (video encoding generates text and vice-versa), and cycle consistency. Apart from these additional supervised losses, unsupervised losses are added with adversarial learning which aim to bring the video and text encoding distributions in the common space close, as well as the standard real and generated video/text distributions close by adding corresponding discriminators (like in GANs). The whole system is learned end-to-end in two phases, first with supervised paired data and then with all paired and unpaired data. The experiments are shown on four datasets: ActivityNet, HMDB, UCF101, MLB-YouTube Positives: - The problem of multimodal learning is an interesting and challenging problem - The paper is novel; while the idea of a joint shared embedding space is not new this paper adds new losses as summarized above and shows reasonably convincing empirical results - The results are shown for diverse applications which highlight the generality of the method - The use of unpaired/unsupervised data is also relatively less explored which this paper incorporates as well - The empirical results given are convincing, eg Tab1 gives a good ablation study showing how the different components affect the performance. SoA comparison are given on a standard task (however see below) Negatives: - Comparison with state of the art result Tab2 should also contain the features used. The performances might just be higher due to the better features used (Kinetics pretrained I3D). Please give a self implemented baseline method with same features but some standard loss in the shared space to give an idea of the strength of the features. - Ideally features provided by previous papers’ authors should be used if available and it should be shown that the proposed method improves results. Overall the paper is well written and had novel aspects which are convincingly evaluated on challenging and diverse tasks.
assistant
 Summary: The paper aims to learn a common embedding space for video appearance and text caption features. The learned shared embedding space then allows multiple applications of zero-shot activity classification, unsupervised activity discovery and unseen activity captioning. The method is based on two autoencoders which have a common intermediate space. The losses optimized encourage the standard unimodal reconstructions in the AEs, along with joint embedding distances (appearance and text of the same video are encoded close by) as well as cross domain mapping (video encoding generates text and vice-versa), and cycle consistency. Apart from these additional supervised losses, unsupervised losses are added with adversarial learning which aim to bring the video and text encoding distributions in the common space close, as well as the standard real and generated video/text distributions close by adding corresponding discriminators (like in GANs). The whole system is learned end-to-end in two phases, first with supervised paired data and then with all paired and unpaired data. The experiments are shown on four datasets: ActivityNet, HMDB, UCF101, MLB-YouTube Positives: - The problem of multimodal learning is an interesting and challenging problem - The paper is novel; while the idea of a joint shared embedding space is not new this paper adds new losses as summarized above and shows reasonably convincing empirical results - The results are shown for diverse applications which highlight the generality of the method - The use of unpaired/unsupervised data is also relatively less explored which this paper incorporates as well - The empirical results given are convincing, eg Tab1 gives a good ablation study showing how the different components affect the performance. SoA comparison are given on a standard task (however see below) Negatives: - Comparison with state of the art result Tab2 should also contain the features used. The performances might just be higher due to the better features used (Kinetics pretrained I3D). Please give a self implemented baseline method with same features but some standard loss in the shared space to give an idea of the strength of the features. - Ideally features provided by previous papers’ authors should be used if available and it should be shown that the proposed method improves results. Overall the paper is well written and had novel aspects which are convincingly evaluated on challenging and diverse tasks.