I was also wondering if PyTorch is passing the right parameters to take advantage of the Tensor Cores, so I added debug statements in the PyTorch code, compiled from source, and ran it.  From the looks of it, PyTorch was passing the right parameters to CuDNN and CUDA to take advantage wherever possible.
I also tried to use nvprof to capture the actual CUDA calls made.  Even though it captures all the CUDA API calls made, it does not show if the execution was done on Tensor Cores or not (as @ngimel mentioned, specific conditions must be met, etc., even though API calls are made with the right parameters/algo.)  I’ve filed a ticket on NVIDIA and they confirmed that they still need to expose such metrics (they are capturing the info, just not exposing) and that they are working on it.