Ok, I’ve confirmed the issue. The fix for now is to reshape the inputs as @kim.seonghyeon said, or to set torch.backends.cudnn.enabled = False. Still, the first approach is likely to be faster for you.
With cudnn disabled the training runs at 1k samples/sec for me. With it slows down to 12.