The real issue is that in topk an invariant condition failed. It would be very useful if you can

run with CUDA_LAUNCH_BLOCKING=1

save every inputs to topk to a file (you can overwrite in each iteration)
upload the input that triggers this assert.

Thanks!