Why would you use a 5D convolution? I would suggest a 3D convolution or a 2D + t solution like a 2d-conv-RNN.
If using a 3D convolution you would use f as number of channels, s as depth and h and w as height and width. Adding a batchsize should work out of the box.
If using a 2d+t approach you would have to transpose the f and the s axis, using the f axis as input channels, h and w as dimensions and the s axis as time axis for recurrence.