The error in cuda here is that you index a tensor with an index which is too big for the dimension you selected.
If there are a few hundreds of them, that means that there are a many invalid indices in this indexing (possibly nan values).
One thing you could do is to add some checks in your code before every indexing operation to check that the index values are not too big, like assert indices.max() < to_index.size(index_dim). That will help you see where the error happens first.
For the segfault, there have been some issue with the 0.3.x binaries where the printing of the stacktrace and error message were causing segfault (that is why you don’t see any proper python stacktrace).
If you are using these binaries, you could upgrade to the latest version to remove the segfault and get a proper stacktrace.