It worked normally until epoch 90 in my case, which is weird…

Actually this is quite common as these asserts often come from indexing out of bound.
So maybe a label in your dataset is incorrect. Or you use a learnt value to index something and that ends up generating a value out of bound.

the code was run with  CUDA_LAUNCH_BLOCKING=1

Given that the stack trace points to the l 31 in transformer.py, and not inside the subsequent_mask() function, I think it means that the error either comes from the .size() or the .to() function.
The first one does not access the GPU (GPU Tensor metadata are saved on the cpu side).
So most likely the second one. And this second one is a know sync point for the GPU and so is usually the place where the error is raised in the async case.
I am surprised though that it still points here with CUDA_LAUNCH_BLOCKING=1. Can you double check that you do set this before doing anything else and that it is taken into account? (you can try a simple program that index out of bound on the GPU to make sure it does what you expect).
Also, do you use multithreading or multiprocessing? These error put the GPU in an un-recoverable state and the process needs to be restarted. So another thread doing something bad could lead another thread to through this kind of errors at random places.