I would say this is most likely because cudnn’s default algorithm behavior change when you get something >255.
You can use torch.backends.cudnn.benchmark=True so that cudnn will pick the fastest algorithm depending on your input/hardware. That should smooth things down