Do you know, if the model is set to eval() from the beginning of the training?
If that’s the case and if only dropout and batchnorm layers are used, I believe you could also remove these layers, since they wouldn’t change anything.
I.e. dropout is disabled, so basically acting as an nn.Identity layer. while batchnorm layers are initialized in a way to not change the incoming activation without training (up to floating point precision):
bn = nn.BatchNorm2d(3)
bn.weight
> Parameter containing:
tensor([1., 1., 1.], requires_grad=True)
bn.bias
> Parameter containing:
tensor([0., 0., 0.], requires_grad=True)
bn.running_mean
> tensor([0., 0., 0.])
bn.running_var
> tensor([1., 1., 1.])

x = torch.randn(2, 3, 24, 24)
bn.eval()
out = bn(x)
print((out - x).abs().max())
> tensor(1.6689e-05, grad_fn=<MaxBackward1>)