Thanks for your reply. I tested the distroless image on our GPU with the MNIST benchmark and it has the same performance as the official image.
As you can see from the link to our Dockerfile in my original message that it is essentially just a “pip install torch”. This already (magically) enables compatibility with GPU at runtime.
(Side note: This was not the case for TensorFlow where I had to copy a bunch of CUDA libraries from nvidia/cuda:cudnn-devel.)