Hi @ptrblck I figured out my issue. It was something to do with the indexing in my cuda code. After debugging, the error was gone!
Thank you!!

UPDATE
I realized this is not related to my code at all. This error happens randomly on random epochs. It looks like something to do with cuda cache collector.
I am working on 3D point clouds where I am sampling the point cloud([1024,3]) in some number of points([512,3]).  If I reduce the output number of points to a very low number (say [8,3]) the code works totally fine but my accuracy is significantly compromised.
I am working on a shared server. Is this issue has anything to do with available GPU memory?
@ptrblck