It looks like this might be a bug in torchvision.transforms.functional.rotate() 5, but you can work around it by defining your RandomRotation transform like so: RandomRotation(20, fill=(0,)). The problem is that by default rotate() passes to PIL the three-channel fill value (0, 0, 0) which of course breaks on one-channel images.