Sorry to bother again. Last time I changed the code as suggested and did some experiment for comparison. However, there is still a gap of performance between 2 approaches which I couldn’t figure out why…
Here is the modified code:
feature_dim = 1024
N, C, T, H, W = video.shape  # T=40
clip_pos = [0, 10, 20, 30]

# approach I
feature = torch.empty(N, 4, feature_dim)
for i in clip_pos:
    feature[:, i] = some_model(video[:, :, i:i+10])  # N, feature_dim

# approach II
data_pack = torch.cat([video[:, :, i:i+10] for i in clip_pos])  # N*4, C, 10, H, W
output = some_model(data_pack)  # N*4, feature_dim
feature = output.reshape(4, N, feature_dim).transpose(0, 1)
'''
code for training
'''

And Here is the training&testing curve(The higher the better. The red&green curve corresponds to approach I)

It seems unreasonable that 2 approaches which did exactly the same thing lead to different results. Or is my modified code still wrong?
Hope for your reply!