Yes sure. I have focused my work on reducing the DataLoader footprint. I have found using a list of int was really really expensive : python ints are 28 bytes long. By using a numpy array I can reduce my RAM footprint from 1 GB to 80 MB. I will see if it scales well on a server.
    train_loader = torch.utils.data.DataLoader(
        medfolder.MedFolder(traindir, file_extensions, patch_size, label_map, file_map,
                  transform=Compose([
                       transforms.CenterSquareCrops([25, 51, 75]),
                       transforms.Unsqueeze(0)
                    ])),
        batch_size=args.batch_size, shuffle=True,
        num_workers=args.workers, pin_memory=True)

    val_loader = torch.utils.data.DataLoader(
        medfolder.MedFolder(valdir, file_extensions, patch_size, label_map, file_map,
                  transform=Compose([
                       transforms.CenterSquareCrops([25, 51, 75]),
                       transforms.Unsqueeze(0)
                    ])),
        batch_size=args.batch_size, shuffle=False,
        num_workers=args.workers, pin_memory=True)

The only difference is that the validation dataset is not shuffled. I will give you some updates on this tomorrow. I think I am nearing the end of this problem.
Really amazed by the community support and your work !! 
Thanks !