@ptrblck Hi, Piotr  Glad you replayed!
Could not get torch.cuda.memory_summary() working for some reasons. I have executed the training from one SSH connection and then tried to run python3 -c 'import torch; print(torch.cuda.memory_summary())' from another SSH connection. However, I got all zeroes which is weird:

Screenshot 2021-04-21 at 11.49.261396×1674 171 KB

At the same time, nvtop and nvidia-smi showed that GPU was utilised:

Screenshot 2021-04-21 at 11.57.013692×1566 251 KB
 
Screenshot 2021-04-21 at 11.57.231440×1134 44.8 KB

In any case, I think that it doesn’t seem to be just a nvidia-smi information misleading.
Namely, I checked that my biggest batch size on the current dataset with FP32 is 8 samples. 9 gives OOM errors. When I enable FP16, the max batch_size still seems to be 8 and 9 still gives me OOM errors.
My model is PyTorch vanilla implementation of the RetinaNet (torchvision.models.detection.retinanet_resnet50_fpn + torchvision.models.detection.retinanet.RetinaNetHead) which is applied to a custom image dataset.
Is there another quick way to check that FP16 is really working on my env (to exclude a case when my particular usage of RetinaNet is somehow faulty)?