Hi,
I try to train my model using PyTorch and a AWS EC2 Instance (g3s.xlarge), but there is a common error, the famous RuntimeError: CUDA error: device-side assert triggered
So have a more clear stacktrace, I run my sscript using CUDA_LAUNCH_BLOCKING=1, and there is all the stacktrace I have:
  /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda [](int)->auto::operator()(int)->auto: block: [0,0,0], thread: [44,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda [](int)->auto::operator()(int)->auto: block: [0,0,0], thread: [26,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCCachingHostAllocator.cpp line=265 error=59 : device-side assert triggered
Traceback (most recent call last):
  File "train.py", line 367, in <module>
    backup_every=args.backup_every)
  File "train.py", line 267, in train
    if torch.isnan(loss):
RuntimeError: CUDA error: device-side assert triggered

This error came during the first epoch, or during the test of the first epoch (it depends)
I check my dataset and:



There is no negative number




All the output files are in Yolo format




There is exactly the same number of files and labels



The number of classes I have is 17 (so I use 17 to generate my cfg, and I enter 17 in my .data file)
I can’t identify where is the mistake, do you have any idea ?
Thanks in advance and have a good day !
Kind regards,
Florian