I’m not sure how you would like to expand the shape of all parameters, which would be needed for a 3D model.
Would feeding each frame separately to the CNN and then forwarding these features to an RNN work instead?