Hi, I am relatively new to pytorch and I am trying to get into image semantic segmentation.
I am trying to use DeepLabv3 and finetune for my use case. To start I decided to use a publicly available dataset (Grand Theft Auto - available here)
As you can see from the following Figure (from the paper), I think that the dataset contains 19 classes:

image974×476 38.1 KB

For the finetuning I understood and adapted the following repository: GitHub - jnkl314/DeepLabV3FineTuning: Semantic Segmentation : Multiclass fine tuning of DeepLabV3 with PyTorch
However, when I try to launch the training (with the main_training.py file) with the following command:
CUDA_LAUNCH_BLOCKING=1 python sources/main_training.py ./dataset_GTA ./training_output_GTA --num_classes 19 --epochs 100 --batch_size 8 --keep_feature_extract

I get the following error:
/pytorch/aten/src/ATen/native/cuda/NLLLoss2d.cu:95: nll_loss2d_forward_kernel: block: [4,0,0], thread: [61,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/ATen/native/cuda/NLLLoss2d.cu:95: nll_loss2d_forward_kernel: block: [4,0,0], thread: [62,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/ATen/native/cuda/NLLLoss2d.cu:95: nll_loss2d_forward_kernel: block: [4,0,0], thread: [63,0,0] Assertion `t >= 0 && t < n_classes` failed.
Traceback (most recent call last):
  File "sources/main_training.py", line 112, in <module>
    args_preprocess()
  File "sources/main_training.py", line 108, in args_preprocess
    main(args.data_dir, args.dest_dir, args.num_classes, args.batch_size, args.epochs, args.keep_feature_extract, weight)
  File "sources/main_training.py", line 80, in main
    model_deeplabv3_state_dict, hist = train_model(model_deeplabv3, num_classes, dataloaders_dict, criterion, optimizer_ft, device, dest_dir, num_epochs=num_epochs)
  File "/home/fabrizioschiano/repositories/DeepLabV3FineTuning/sources/train.py", line 90, in train_model
    loss = criterion(outputs, labels)
  File "/home/fabrizioschiano/repositories/DeepLabV3FineTuning/deeplabv3virtualenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/fabrizioschiano/repositories/DeepLabV3FineTuning/deeplabv3virtualenv/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 1150, in forward
    return F.cross_entropy(input, target, weight=self.weight,
  File "/home/fabrizioschiano/repositories/DeepLabV3FineTuning/deeplabv3virtualenv/lib/python3.8/site-packages/torch/nn/functional.py", line 2849, in cross_entropy
    return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
RuntimeError: CUDA error: device-side assert triggered



I am on Ubuntu 20.04.
I know that this is a problem with the number of classes since when instead of the previous command I change num_classes to 35 I don’t have any issue.
CUDA_LAUNCH_BLOCKING=1 python sources/main_training.py ./dataset_GTA ./training_output_GTA --num_classes 28 --epochs 100 --batch_size 8 --keep_feature_extract

(with num_classes to 34 I have the same issue)
I know that the error is being thrown by the following line:
loss = criterion(outputs, labels)

which has a problem when calling this line:
return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)

which is the line after the print("before returning")
I would like to understand how to debug this problem. I think it is a dimensionality problem of the labels but I can’t seem to understand how to fix it.