Instead of the fop loop you could directly use torch.argmax(output, dim=1), but your code should also work.
I guess the correspondence between the data and labels might be broken. Could you check some random examples from your ground truth file and the input image manually?