There are multiple possible approaches, which you could try:

you could represent the sequence of images as a new temporal dimension, execute the feature extractor (conv-pool layers) in a loop, and pass the output to an RNN. Afterwards you could use the activation of the last time step and feed it to a classifier (linear layers).
instead of using the loop in the feature extractor, you could use nn.*3D modules and use the depth dimension as the temporal dimension.
stacking the frames in the channel dimension and using grouped convs could also work, but I think this would

Maybe others have more and better ideas.