Solved by ptrblck in post #2 
              
              
                If a submodule already uses all SMs on your GPU, you won’t be able to run another model in parallel. 
A lot of layers are used in such a kernel launch configuration to saturate the GPU as much as possible.