I have some really big input tensors and I was running into memory issues while building them, so I read them one by one into a .pt file. As I run the script that generates and saves the file, the file gets bigger and bigger, so I am assuming that the tensors are saving correctly. Here is that code:
with open(a_sync_save, "ab") as f:
     print("saved")
     torch.save(torch.unsqueeze(torch.cat(tensors, dim=0), dim=0), f)

I want to read a certain amount of these tensors from the file at a time, because I do not want to run into a memory issue again. When I try to read each tensor saved to the file I can only manage to get the first tensor.
with open(a_sync_save, "rb") as f:
    for tensor in torch.load(f):
        print(tensor.shape)

The output here is the shape of the first tensor, then quits peacefully.

  
      stackoverflow.com
  
  
      
    
  

  Pytorch reading tensors from file of tensors 5



  python-3.x, pytorch, torch



  asked by
  
  
    Race
  
  on 03:17AM - 26 Sep 20 UTC