If I had to guess it definitely looks more like a pytorch problem. I’m having a little trouble running your code but I’ll try again later today and report back.
One thing that you could try in the mean time is building pytorch from source and running the code on a GPU with that. There’s been a couple of fixes to memory errors and this could be one of them.