One thing to know is that once a GPU sends an assert, it goes in a bad state and the whole process needs to be restarted, in your case, you need to restart the kernel from scratch. Otherwise, you will see random errors every time you try to use the GPU (this is a limitation on CUDA side).
You should double check also is that every place where you using indices (targets for loss, or when indexing), check that these indices are in the proper range.
If both of these are done, you will have to remove code to try and pinpoint the reason for the failure I’m afraid.