I looked into the issue and it’s a problem with our code that chooses the cuDNN algorithms. PyTorch is faster at first, but then cuDNN asks for 17GB of mem, and we just fall back to the slowest algo because we can’t satisfy that. It should be fixed soon. Thanks for the report and code that reproduces it!