Solved by ptrblck in post #2 
              
              
                AMP shouldn’t use more memory and I assume you are trying to use the global batch size in each process and thus GPU. 
As explained here you should set the batch_size for each GPU as the local batch size (by dividing the global batch size by the number of GPUs).