So the performance gap persists when I compare outside the docker container, I didn’t realize that the difference in the hardware was the issue. The Titan is much older than the 1080 Ti.
GeForce GTX 1080 Ti
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
324    8.589    0.027    8.589    0.027 {method ‘cuda’ of ‘torch._C._TensorBase’ objects}
GeForce GTX TITAN
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
324   21.860    0.067   21.860    0.067 {method ‘cuda’ of ‘torch._C._TensorBase’