Ok, @colesbury made a good point. The fix is to add
torch.backends.cudnn.benchmark = True

This is a flag that controls cudnn benchmark mode and should be used only if every forward pass uses the same data dimensions. After this your code runs @ 7k/s for training and 64k/s for test on my machine.