MARGINALIZED AVERAGE ATTENTIONAL NETWORK FOR WEAKLY-SUPERVISED LEARNING
In weakly-supervised temporal action localization, previous works have failed to locate dense and integral regions for each entire action due to the overestimation of the most salient regions. To alleviate this issue, we propose a marginalized average attentional network (MAAN) to suppress the dominant response of the most salient regions in a principled manner. The MAAN employs a novel marginalized average aggregation (MAA) module and learns a set of latent discriminative probabilities in an end-to-end fashion. MAA samples multiple subsets from the video snippet features according to a set of latent discriminative probabilities and takes the expectation over all the averaged subset features. Theoretically, we prove that the MAA module with learned latent discriminative probabilities successfully reduces the difference in responses between the most salient regions and the others. Therefore, MAAN is able to generate better class activation sequences and identify dense and integral action regions in the videos. Moreover, we propose a fast algorithm to reduce the complexity of constructing MAA from O(2 T ) to O(T 2 ). Extensive experiments on two large-scale video datasets show that our MAAN achieves a superior performance on weakly-supervised temporal action localization.
In this paper the authors focus on the problem of weakly-supervised action localization. The authors state that a problem with weakly-supervised attention based methods is that they tend to focus on only the most salient regions and propose a solution to this which reduces the difference between the responses for the most salient regions and other regions. They do this by employing marginalized average aggregation to averaging a sample a subset of features in relation to their latent discriminative probability then calculating the expectation over all possible subsets to produce a final aggregation.
The problem is interesting, especially noting that current attention methods suffer from paying attention to the most salient regions therefore missing many action segments in action localization. The authors build upon an existing weakly-supervised action localization framework, having identified a weakness of it and propose a solution. The work also pays attention to the algorithm's speed which is practically useful. The experiments also compare to several other potential feature aggregators.
However, there are several weakness of the current version of the paper:
- In parts the paper feels overly complicated, particularly in the method (section 2). It would be good to see more intuitive explanations of the concepts introduce here. For instance, the author's state that c_i captures the contextual information from other video snippets, it would be good to see a figure with an example video and the behaviour of p_i and c_i as opposed to lamba_i. I found it difficult to map p_i, c_i to z and lambda used elsewhere.
- The experimental evidence does not show where the improvement comes from. The authors manage to acheieve a 4-5% improvement over STPN through their re-implemenation of the algorithm, however only have a ~2% improve with their marginalized average attention on THUMOS. I would like to know the cause in the increase over the original STPN results: is it a case of not being able to replicate the results of STPN or do the different parameter choices, such as use of leakly RELU, 20 snippets instead of 400 and only rejecting classes whose video-level probabilities are below 0.01 instead of 0.1, cause this big of an increase in results? There is also little evidence that the actual proposal (contextual information) is the reason for the reported improvement.
- There seems to be several gaps in the review of current literature. Firstly, the authors refer to Wei et al. 2017 and Zhang et al. 2018b as works which erase the most salient regions to be able to explore regions other than the most salient. The authors state that the problem with these methods is that they are not end-to-end trainable, however Li et al. 2018 'Tell Me Where to Look': Guided Attention Inference Network' proposes a method which erases regions which is trainable end-to-end. Secondly, the authors do not mention the recent work W-TALC which performs weakly-supervised action localization and outperforms STPN. It would be good to have a baseline against this method.
- The qualitative results in this paper are confusing and not convincing. It is true that the MAAN's activation sequence shows peaks which correspond to groundtruth and are not present in other methods. However, the MAAN activation sequence also shows several extra peaks not present in other methods and also not present in the groundtruth, therefore it looks like it is keener to predict the presence of the action causing more true positives, but also more false positives. It would be good to see some discussion of these failure cases and/or more qualitative results. The current figure could be easily compressed by only showing one instance of the ground-truth instead of one next to each method.
I like the idea of the paper however I am currently unconvinced by the results that this is the correct method to solve the problem.