Thanks for the code. I cannot see anything obviously wrong.
Could you create a simple DataLoader loop with num_workers=0 and num_workers>=2 and compare the memory usage via:
loader = Dataloader(dataset, num_workers=...)   # use 0 and >=2 in the next run
print(torch.cuda.memory_allocated()/1024**2)

for images, targets in loader:
    images = images.cuda()
    targets = target.cuda()
    print(torch.cuda.memory_allocated()/1024**2)

and post the result please?
PS: Variables are deprecated since PyTorch 0.4 and the usage is .data is discouraged, as it might have unwanted side effects. You can use tensors instead of Variables and call loss.detach(), if you want to store the tensor without the computation graph.