I do normalize my heatmaps while training though.
If you remove the softmax loss the loss will start go down though. In the paper they mention about using a linear activation before the softmax, I am assuming it is doing nothing in PyTorch

throws back an error
heatmap = heatmap/torch.sum(dim=(2,3,4), keepdim=True)


Sorry to keep adding info along the way. I thought it could a helpful reference to rule out or consider new possibilities