NaN values are tricky to handle, as you cannot simply mask them e.g. by multiplying with zeros.
If you are using a mask for your targets I would recommend to use a specific class index (e.g. nb_classes+1) and try to deal with these values instead.