Solved by mcarilli in post #5 
              
              
                I don’t think the apex amp API supports this without complex/undocumented hacks.  Apex amp is in maintenance mode now, no new features will be added. 
However, torch.cuda.amp is designed to support model parallelism (ie different layers on different device) out of the box.  Please consider upgrading…