In that case you can use torch.argmax instead with same config. It will return the proper indices of channels instead of probs/logits.
About your solution, the computational graph is broken I think due to conversion to numpy and forcing loss to have gradients (use .requires_grad instead of Variable as it is deprecated) will not give you the correct gradients.