Miguel, for this 1 by 2 tensor, is its value continuous (like [0.2, 0.35] is allowed) or discrete [0, 1], [1, 0], [0.5, 0.5]?
If it is continuous, then MSE might be better. If it is discrete, then NLL is better, you can actually just do 0, 1, 2 label for those 3 discrete values.
BTW, you might want to check the sample distribution as well. Example: if you have discrete values, but your training data is mostly [0, 1], and very few [1, 0], this imbalance could cause cost value to saturate. You can mitigate this by using a weight tensor inside nll_loss.