Since you are also changing the input shape (from the original 4-dimensional inputs to 5-dimensional ones), you would need to change at least the first conv layer (e.g. by replacing it with nn.Conv3d, which accepts 5-dimensional inputs). However, I’ve mentioned “at least” since the output of this new conv layer would also be 5-dimensional, so if you want to keep this activation shape you would need to replace the entire feature extractor.
I’m not familiar with your use case, but maybe iterating the input could also work, i.e. indexing the input in the temporal dimension, passing each slice to the model, and concatenating it afterwards before feeding it to the classifier.