I want a derivative with respect to the source tensor[index] -> the tensor on the “index” location.
Because the output tensor of FCSN 2 architecture is in the shape of [1,2,1,#frame]. This tensor means whether frames are selected or not.
The algo. of this paper 2 is below:


Downsampling every video in 2fps and pre-processing every downsampled training video to [1,1024, 1, T] (video) and [1,1024,1,S] (summary) through pre-trained googlenet.


for every pre-processed downsample video’s feature(in the format [1,1024,1,T]) and real summary video’s feature(in the format [1,1024,1,S]):  ->T,S may differ in each video



Put [1,1024,1,T] into FCSN and get the index_mask(this index_mask is constructed from the output of FCSN [1,2,1,T] which means which frame should be selected)

S__44130656.jpg1478×1108 190 KB






Select K key outputs of FCSN according to index_mask and get the output in format [1,2,1,K].




Put the selected K key outputs of FCSN([1,2,1,K]) into the 1x1 conv to get the [1,1024,1,K].




Add K key features[1,1024,1,K] (Pick the K key features from original video feature according to index_mask) to the 1x1 conv output[1,1024,1,K] to do skip connection(this is the output of S_K).




Pick the K key features from original video feature according to index_mask and calculate the reconstruction loss with previous step output.




Calculate the diversity loss.




Calculate the adv. loss by putting the output of S_K in to S_D and set target score 1 to get the adv. loss




Update S_K.




Put Real summary videos’ features [1,1024,1,S] into S_D to calculate and set target score 1 to get the adv. loss.




Put Fake summary videos’ features [1,1024,1,K] come from S_K in S_D to calculate and set target score 0 to get adv. loss.




Update S_D.



end


Thank you very much, Tony-Y.