OK so I cleaned up my code a bit and it seems to work fine now.
I had a random process that was built upon many Pytorch exceptions that I catch until what I want works. Changed it to something cleaner that doesn’t rely on exceptions. Will update if message appears again in the run.