You are doing it right. Proper order is BxCxSXHxW.
3Dconv are really data greedy and they tend to overfit. You should have a really huge dataset to obtain good results. You can have a look at this paper.

  
      
      arXiv.org
  
  
    

A Closer Look at Spatiotemporal Convolutions for Action Recognition 5

In this paper we discuss several forms of spatiotemporal convolutions for
video analysis and study their effects on action recognition. Our motivation
stems from the observation that 2D CNNs applied to individual frames of the
video have remained...


  
  
    
    
  
  


They create networks modeling motion with 3dconvs at the beginning and 2d conv in the end.
BTW image2image translation is performed with encoder-decoder architectures, what are you exactly using?