Solved by ebarsoum in post #2 
              
              
                N is the batch size, the number of sample. In your case, sample != single frame. Your sample is HxWx1x20., (c=1, you have only one channel) . 
To map that to (N,C,D,H,W). C=1, D=20, H=H, and W=W. N depends on your batch size, and if it fits in memory or not.