Solved by Caruso in post #2 
              
              
                Hi @Achu_Chandran, 
I’m not hundred percent sure, if I understood the data shapes correct, but you have a sequence of 100 images, which have 3 channels and are of shape 16 x 701? 
In this case you could use a  Conv3d for the temporal reduction and another Conv3d or Conv2d for the spatial operation. 
…