RuntimeError: cuda runtime error (59)
I would greatly appreciate any pointers.
kundan2510
I had an issue with the embedding layer. I had an index which was greater than the vocab-size. Fixing that resolved the issue. I still don’t understand how the error shown is related to that.
kundan2510
Because of the asynchronous nature of cuda, the assert might not point to a full correct stack trace pointing to where the assert was triggered from.
smth
STORY TIME! I recently re-installed my NVIDIA drivers (someone in the lab broke the server trying to install them, and for the life of me I have no idea what went wrong) and managed to compile the latest PyTorch build from source (which previously had thrown errors related to CUDA, suggesting that PyTorch unsurprisingly doesn’t compile if you’re using old drivers). Yay! It turned out the issue was in the line above, where I call index_select on a different tensor using a variable that can vary widely–the issue was that I clamped my indices incorrectly: out = inp.index_select(2, torch.clamp(index, 0, inp.size()[-1])).
ajbrock
smth
Or is the tensor.index_select function just something that’s hard-baked into CUDA that PyTorch doesn’t actually wrap? I might be misunderstanding how it’s put together.
ajbrock
apaszke
Sorry, I was just trying to do that as an example of indexing outside the allowable array size. Here’s a better example: returns RuntimeError: out of range at /data/users/soumith/miniconda2/conda-bld/pytorch-0.1.7_1485444530918/work/torch/lib/TH/generic/THTensor.c:379 Is it not possible to insert a value check in the index_select function to ensure that the elements of the second argument are all less than the size of the indexed tensor along that axis?
ajbrock
It’s impossible, because we’d have to execute max and min over the whole tensor, synchronize the CPU with the GPU, and execute the index_select kernel only if the values are correct. It would be too slow.
apaszke
Got it, that makes sense.
ajbrock
This help me, thanks.
11129
same here, thanks for the hint… I was converting some code from torch to pytorch. Lua is 1 based index but Python is 0 based index. My dataset was created for lua hence it is off by one.
euwern
I had this error when have created the model with the wrong number of classes:
gleb
miladiouss
I faced with the problem that is very similar: ‘reduce failed to synchronize: device-side assert triggered’,and the problem occurred in the circumstance that I used the BCEloss.finally I found the cause that my output is a mat consists of the elements that is negative numbers,but the implement of BCEloss is about log(),which has no meaning in the negative area, thus I add a sigmoid to constrain the output between 0-1and my problem is solved.
Fengzi
Thanks
jdenim
m0nster
I’m not sure if I’m missing the obvious, but how about subtracting 1 from the labels? Best regards Thomas
tom
m0nster
tom