There’s a multi-GPU torch.cuda.amp.GradScaler test case 6 that ensures ordinary GradScaler usage supports networks with layers on different devices.
torch.cuda.amp.autocast locally enables/disables autocast for all devices used by the invoking thread.  (However, the autocast state is thread local, so if you spawn a thread to control each device, you must re-invoke autocast in the side thread(s).  This affects usage with torch.nn.DataParallel and torch.nn.parallel.DistributedDataParallel with multiple GPUs per process 4.)