Hi @Achu_Chandran,
I’m not hundred percent sure, if I understood the data shapes correct, but you have a sequence of 100 images, which have 3 channels and are of shape 16 x 701?
In this case you could use a  Conv3d 2 for the temporal reduction and another Conv3d or Conv2d for the spatial operation.
Here is an example:
B, S, C, H, W =  2, 100, 3, 16, 701
HIDDEN_C = 8        # not sure if you need a hidden channel

t = torch.randn(B, S, C, H, W)
t = t.transpose(1, 2)   # swap seq and channels dim -> SHAPE: [B, C, S, H, W]

conv_temp = nn.Conv3d(3, HIDDEN_C, kernel_size=(100, 1, 1))
conv_spatial = nn.Conv3d(HIDDEN_C, 3, kernel_size=(1, 3, 3), padding=(0, 1, 1))

out = conv_spatial(F.relu(conv_temp(t)))
out = out.squeeze(2)    # remove seq dim -> SHAPE: [B, C, H, W]

print(out.shape)

If I understood your data shapes wrong, please correct me.
Greetings.