Could you please clarify your design a liitle bit more? your description of “distributed” and “decomposing” is pretty vague.
Is it a multi-process application? How do you map your gpus to your processes? How do you map your model to your devices? I am really sorry that I cannot help you more if you don’t give a clear definition of the architecture you would like to achieve. It would be better if you can draw a graph or show your code.
If you just want to split layer1, layer2 and layer3 to different devices, you can simply save them individually with torch.save and torch.load, torch will take the charge to pickle whatever passed to it, including parameters and your custom attributes such as the ones set by self.some_attr in __init__.