Hello Sir,
I realized that this was due to my higher batch size. Reducing batch size from 32 to 16 removed this issue.
Thank you!

EDIT
I thought reducing batch size solved the issue, but I was wrong.
I reduced the batch size to 4 and ran my code. The code worked fine for initlal 4-5 batches and then runs to “notorious”
RuntimeError: CUDA error: an illegal memory access was encountered
I have a custom function built using cuda and pytorch cpp binding.
The custom function works well for initial few batches and then fails.
I don’t know if I will be able to share a workable snippet. But I have the library files and a test notebook which I can share in case you need to reproduce the error.
Thank you!