Another thought: note that having 4 submodules and 4 GPUs won’t use one submodule per GPU, if that’s what you intended (unless perhaps only one module is selected and used in a forward pass).
What is your nvidia-smi output when running? What is your device variable?