It seems strange that one work flow seems to work on Colab, while another fails.
Are you also restoring the optimizer.state_dict() in both use cases?
Could you run a small test and compare the output for a fixed input, e.g. torch.ones()?
If you get exactly the same outputs, I would guess that the way you are restoring the data loading pipeline might create the difference.