Solved by spanev in post #2 
              
              
                Hi @Steve_Hu, 
You can first try to zero pad the input: 
# Given input = (B, 2, T, H, W)
input = torch.cat((input, torch.zeros(B, 1, T, H, W)), axis=1)

You can also modify the first layer and fine tune it with your 2 channels input. To modify it: 
net = torchvision.models.video.r2plus1d_18()
r2.ste…