@ptrblck Found what is causing the error. It is the Sparse optimizer. When I do not use a sparse embedding layer or optimizer, there seem to be no problem at all. Do you know any such issues relating to sparse optimizer?
@smth Hi Soumith, Have you seen sparse optimizer causing CUDA error. I am guessing it is caused by  accumulating params inside  GPU inside training loop? I highly appreciate any help in the correct direction. Thanks a lot!