I had a similar issue with a TCN network: semi-random CUDA error after a random number of steps.
I probably have a bad gpu, tough I still not 100% sure. I would say I’m 90% sure the GPU is faulty, but that GPU can still train RNN models fine…
My suggestions:

If you find a way to trigger the error faster (ideally since the first batch), that would help tremendously for debugging. In your training runs where the error happened faster, what did you change specifically (batch size, regularisation, architecture?), try fiddling with those params to get the error to trigger faster.
If you have the time/know-how, you could replicate your setup on google cloud (or colab) and by renting a GPU.
Try simplifying your architecture (removing all convs, removing max pooling, removing all relus) and see if the error goes away. The issue is that without point 1., this will take a long time for each run.
Create a “reproduction dump” where you provide a gist or repo with instructions so that it’s easier to find someone willing to reproduce (or not) your issue.
Try giving the network super small inputs, and super large inputs that barely fit on your GPU, and see if the timing of the error changes. If it does, it’s more likely linked to a memory issue. Monitor the run with nvidia-smi -l 1