Dear all,
I am running into a problem when training a segmentation net on Pascal VOC using multiple GPUs (DistributedDataParallel). Pytorch seems to expect targets to have label values only between 0 and n_classes–1 (inclusive). However, this does not consider the presence of the ignore label (whose value is often 255).
More specifically, I get the error:
aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: Assertion t >= 0 && t < n_classes failed.
In my case, n_classes = 21 (Pascal VOC 2012) and my targets have label values in [0, 20] + [255] (for the ignore label). I ignore the 255 label in the CrossEntropy cost function. How do I get targets with ignore labels to pass the assertion test?
Interestingly, I don’t get this assertion failure when DistributedDataParallel is not used.