Hi Avi!
In addition to @ptrblck’s suggestion, you could try using the
class-weights feature of CrossEntropyLoss.  This allows to
to give each class its own weight in the overall loss.  For your
use case I would imagine setting the weight of your “I don’t care”
class to zero and the weights of your “real” classes to one.
Note, both this and @ptrblck’s approach entail calculating the
loss for all of the pixels, and then discarding most of the calculated
losses (by multiplying them by zero).  This may seem inefficient
if most of your pixels are “I don’t care,” but, presumably, when an
image is pumped through your network, (almost) all of the input
pixels affect the per-pixel predictions for the “real” pixels, so the
overall network processing dominates the cost of calculating the
loss.
Good luck.
K. Frank