Hi All,
I am training ResNet on CIFAR-10 dataset and doing TenCrop data augmentation as ResNet suggests. However, I found that after TenCrop (please see the implementation of my dataset below), 10 pictures cropped from the same picture will be glued together in a batch even I turn on shuffle=True. I am wondering if there is any way to solve this (i.e. to treat 10 pictures independently as we normally do for training pictures: crop 1 in batch i, crop 2 in batch j,…, instead of crop 1-10 all in the same batch)? Or for the training, this does not matter? I thought this problem matters since glueing all cropped pictures will decrease the randomness/variance of gradients which is more likely to be overfitting according to the second answer of this thread, but I am not sure.
My implementation:
class CIFAR10(Dataset):
    def __init__(self, data_path, dataset, data_aug=False):
        """
        data_path: folder storing train, valid and test set
        dataset: "train", "valid" or "test"
        data_aug: if use data augmentation, default: False
        """
        # initialize object variables
        self.data_aug = data_aug

        # read in the dataset
        with open(f"{data_path}/{dataset}_set.pkl", "rb") as fin:
            self.cifar_imgs = pickle.load(fin)

        # define transform
        # we keep the original image size 32*32, same as experiments in ResNet paper
        # Note that after ToTensor, the value of each pixel becomes [0,1]
        # then we can apply Normalize
        if data_aug:
            # Here the data augmentation is based on https://arxiv.org/pdf/1409.5185.pdf
            # Also, please take a look at PyTorch doc about how to solve dimension
            # problems due to tuples returned by TenCrop()
            self.transformations = transforms.Compose([
                transforms.Pad(padding=4),
                transforms.TenCrop(32, vertical_flip=False),  # return a tuple of 10 PIL images
                transforms.Lambda(lambda crops: torch.stack([transforms.ToTensor()(crop) for crop in crops])),  # convert the tuple into [B, C, H, W] 
                transforms.Lambda(lambda tensors: torch.stack([transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616))(t) for t in tensors]))
                ])  # Note: valid and test should not use data aug
        else:
            self.transformations = transforms.Compose([
                transforms.ToTensor(),
                transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616))
            ])    
    
    def __getitem__(self, index):
        img = Image.fromarray(self.cifar_imgs[index][0]) # convert to PIL image
        label = self.cifar_imgs[index][1]
        
        # transform the image
        img = self.transformations(img)

        return img, label

    def __len__(self):  
        # notice that we cannot return the length of the list after data augmentation
        # otherwise, the sampler will sample from 1 to length after data augmentation
        # which will cause out of range error when getting items
        return len(self.cifar_imgs) 

Also, I used the trick in the PyTorch doc to deal with the inconsistency of dimension (4D vs. 5D) as follows:
if if_aug:
   bs, ncrops, c, h, w = img.size()
   img = img.view(-1, c, h, w)
   label = torch.repeat_interleave(label, 10)