i figured out the NaN reason ,i was using mixed FP 1e-12 was rounding to zero so i changed that to 1e-7. that got resolved.
For the difference part
objective is to predict vehicle or object presence on road on certain spatial location coordinates from Driverless cars’ camera point of view.
While creating label i created Mask of shape X,Y  whose non zero pixel  corresponds to 2d pixel position in camera space projected from its 3d position on road  .Basically model will detect car at position based on masks non zero pixel.
Custom one gives values in thousands but comes down fast ,standard one gives 0 to 1 and stagnates after 0.01.