Setting NVIDIA_VISIBLE_DEVICES to a GPU id makes that GPU available exclusively for the docker container. A GV100 has 5120 CUDA cores and 32GB HBM2 graphics memory.
What I want to be able to do is to allocate 2560 CUDA cores and 16GB of graphics RAM, effectively splitting the GV100 resources into 2, and allocate 2 docker containers to it and get LMS to work. You could consider going a bit more aggressive and allocate say 1280 CUDA cores and 8GB of graphics RAM and allocate 4 docker containers to a single GPU and try to get LMS to work.
This requires virtualization of GPU resources.
As I understand virtualization of GPU resources is only supported by VMware for x86 platforms. VMware is currently not available for Power9. For the moment if you run Ubuntu or RHEL on the AC922, the only option is to running multiple docker instances is to make a dedicated GPU resource available and pass the entire GPU up to the docker container. This will restrict the maximum number of docker containers to 4 on an 8335-GTH, and 6 on a 8335-GTX.
For a reinforcement learning context, I want to run between 8 to 16  docker instances with GPU acceleration to maximize distributed agent training within a single node, before scaling out horizontally across multiple AC922 nodes.