Unfortunately, this is expected for some cases where cudnn heuristics is wildly wrong. benchmark = False deterministic=True uses default algorithm (implicit precomp gemm, iirc), benchmark = False deterministic = False (vanilla) calls cudnn heuristics to pick an algorithm, which may or may not be better than always using default.