I’m not sure, how your prediction exactly looks like.
Are you trying to predict new video frames or what do you mean by motion?
Could you post a dummy prediction?
I think PyTorch will meet your needs. Have a look at the tutorials 72 to get familiar with the framework.
It’s possible to combine input frames and it depends on the use case.
You could combine them in the depth dimension, i.e. you would have an input of [batch_size, channels*2, h, w], although I’m sceptical it’s the most useful approach here.
Another way would be to combine them side by side, i.e. your input would be [batch_size, channels, 2*h, w].
Also, you could use the frames separately for some layers and then combine the activations later in the model.