In case you are using e.g. batchnorm layers, the dummy forward pass should be performed in model.eval() mode. Otherwise the fake inputs would be used to update the running stats of all batchnorm layers.
Could this be the case?