TheeNinja:

My understanding of mixed precision training is that there is a tensor of master weights that is FP32. Each iteration of training, a local copy of FP16 weights is made, which is used to do the forward-propagation and back-propagation.


Master parameters and gradients are used in apex/amp with opt_level='O2'.
The current PyTorch master (and the nightly binaries) contain the native mixed-precision implementation, which we recommend to use now, and you can have a look at the documentation here 7.



 TheeNinja:

If all the propagation and matrix multiplications (done on the GPU) only use this local copy of FP16 weights, does that mean that the FP32 master weights can be stored in CPU memory, with the FP16 weights being a copy from CPU memory -> GPU memory (that also halves precision) performed each iteration?


While that might be possible, you would most likely lose the performance benefits from using TensorCores due to the data transfer, so the master params are stored on the GPU.



 TheeNinja:

By explicit, I mean whether I should be calling half() on self.x_batch and self.y_batch (even though my Trainer is already with precision=16 ), and whether I should call to(my_cuda_device) or not even though my Trainer was provided with gpus=1 . In other words, what will be taken care of of vs. what am I supposed to do on my end explicitly?


You should not call half() on the data or the model and apex/amp or torch.cuda.amp.autocast take care of the casting for you.
I’m not sure where the Trainer class comes from, but I assume it’s from a high-level API?
If so, I would expect the same behavior, i.e. no half() calls.



 TheeNinja:

Finally, how do the pin_memory and num_workers DataLoader arguments change up things here?


These arguments to the DataLoader are completely independent from mixed-precision training.
num_workers will use multiprocessing, where each worker loads a batch in the background while your training is running.
pin_memory=True will use page-locked memory to speed up the data transfer between the host and device and allows you to use tensor.to(non_blocking=True) to an asynchronous transfer.