Thanks for the update.
So back to the first question. You mentioned that you are only running out of memory, if no annotations are found for the current image? I assume this is a property of some images?
If that’s the case, let’s take a look at the posted FocalLoss implementation and search for differences between the different paths for annotated images and one without annotations.