What is the most efficient way to do a multi batch prediction in PyTorch?
I have a bunch of images (Dogs vs Cats test set to be precise) that I want to run prediction on. I call the following code in a loop over Dataloader Iterator with a batch size of 64 and store the result int a torch tensor. How should I efficiently collect all the results on the GPU and transfer it to host?
# Loop over 
def step(self, inputs):
    data, label = inputs # ignore label
    outputs = self.model(data)
    _, preds = torch.max(outputs.data , 1)
    # preds, outputs  are cuda tensors. Right?
    return preds, outputs

def predict(self, dataloader):
    for i, batch in enumerate(dataloader):
        pred, output = self.step(batch)
        # How to collect these results efficiently without incurring performance penalty ?