After broadcasting, you end up having a matrix of 4000 x 32 x 35000 which is more than 16 GiB which could not be saved even in the first step. I tried smaller batch sizes like 6 and it worked but you may need to check https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html.