Are you using the Nvidia Container Toolkit? If not, I think this may resolve the issue for you. Please let me know how it works out.
How to Properly Use the GPU within a Docker Container | by Jacob Solawetz | Towards Data Science 33