My current implementation is using pytorch 0.4 with python 3.6 and I am facing the issue that the computation time of the loss (nn.MSE) is constantly increasing if I enable pinned_memory in the data loader.
I.e the first batch requires 120 ms of computational time, the second requires 270 ms and all remaining batches require 630 ms.
After num_workers batches have been processed, the computational time drops down to 120 ms and increases in the same manner.
Beside enabling pinned memory in the data loader, I manually pin the memory of all tensors contained in the batch and move them to GPU as follows
gt = sampled_batch['gt'].pin_memory().cuda(device=self.cuda_device, non_blocking=True)
mask = sampled_batch['mask'].pin_memory().cuda(device=self.cuda_device, non_blocking=True)
input = sampled_batch['image'].pin_memory().cuda(device=self.cuda_device, non_blocking=True)
input_var.requires_grad_()

For testing purpose, I simply calculate the loss as follows and measure the time
start = time.time()
loss = self.criterion(estimate, gt) 
print("Calculation time %f" % (time.time() - start))

Disabling pinned memory would solve the issue for the loss calculation with the downside of increasing the required time to copy the batch to GPU within each iteration.
What am I missing here?
Thanks in advance