I’m trying to process some videos by dividing each video into 4 clips(video length=40, clip length=10) and pass them through the model.
Here is the code:
feature_dim = 1024
N, C, T, H, W = video.shape  # T=40
clip_pos = [0, 10, 20, 30]

# approach I
feature = torch.empty(N, 4, feature_dim)
for i in clip_pos:
    feature[:, i] = some_model(video[:, :, i:i+10])  # N, feature_dim

# approach II
data_pack = torch.cat([video[:, :, i:i+10] for i in clip_pos])  # N*4, C, 10, H, W
output = some_model(data_pack)  # N*4, feature_dim
feature = output.reshape(N, 4, feature_dim)  # output.view(N, 4, feature_dim) lead to the same bad result

'''
code for training
'''

The approach I works fine, but since it uses for loop which is less efficient, I tried the approach II to feed forward all the data at once. However it does not perform well in training and I dont know why.
I guess there may be something tricky happened when performing concatenate & reshape on tensor that make gradients flow abnormal.
How can I modify approach II to make it work? And why do the 2 approaches lead to different results?
Thanks!