The model execution shouldn’t be slower on Pascal GPUs but you also shouldn’t expect significant speedup. The model execution should benefit from the reduced memory transfers, which could show a speedup even on older GPUs.
Are you seeing the slowdown on the 1080Ti even with torch.backends.cudnn.benchmark=True?