I solved it.
The statement os.environ['CUDA_LAUNCH_BLOCKING'] = "1" needs to be executed before even loading torch. Then it helps give a better stack trace of error.
In my case, the error was when the captions were fed in the embedding layer in decoder. I had defined the vocab size to be four words short and when those words were encountered, the embedding layer couldn’t handle those.