I had the same problem, because when I forwarded the ConfusionMatrix metric (metric()) at every training step it hasn’t had encountered every class yet, so it couldn’t properly normalize the counts (dividing zero by zero). I solved it by only updating the metric (metric.update()) at every training step, and since I log only every 50 training steps where every class has been encountered by then, I have no warnings anymore.