I think data augmentation would help a lot in your case. It’s not too difficult to add either, for example you could do something like this:
my_transforms = transforms.Compose([
    transforms.ToPILImage(),
    transforms.ColorJitter(brightness=0.5), # random brightness
    transforms.RandomRotation(degrees=45), # max degree 45
    transforms.RandomHorizontalFlip(p=0.5), # flips horizontal with prob 0.5
    transforms.RandomGrayscale(p=0.2), # converts to grayscale with prob 0.2 (keeps channels)
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.0,0.0,0.0], std=[1,1,1]) # Note: this does nothing!
    ])

my_dataset = datasets.MNIST(root='dataset/', train=True, transform=my_transforms, download=True)

There are a lot more transforms you could use and you can read more about them here: https://pytorch.org/docs/stable/torchvision/transforms.html 5. Also depending on what images you have it might not make sense to have certain transformations. Like in our case with MNIST dataset, RandomHorizontalFlip() or RandomVerticalFlip() would probably not make too much sense.