Hi,
I am trying to run over 6000 custom layers initialized with nn.ModuleList() on GPUs. I can use 80-120 GPUs on cluster. What is the best way to distribute the layers on the GPU? Should I do it in init method or in forward? Do the layers run asynchronously when I execute them in a for loop? I don’t know how to execute over 6000 layers more efficiently with pytorch.
Thank you very much for your help