How did you split the model for the first runs?
Did you manually “chopped” the model in different parts and pushed each to a different device?
If so, try to move more layers/submodules to other devices.
I’m not sure, how the Architect[(20, 8), (12, 4, 3)] class works exactly, but I assume it’s creating different submodules?