I don’t think you should shuffle the images if you want to leverage their temporal order. By shuffle I mean in the 3D axis, so I would load images like this batch 0, [Id, Id+1 …], 3, 256, 1024. But if train with more than one batch size than I would shuffle on the 0 axis while training.
Changing the 2D layers to 3D you will increase the number of parameters by about a factor of 3. Make sure you have enough training data before you try.
Also, look at keypoint rcnn first, it might be enough for your application.