reudrax:

I am working on a gpu cluster, sometimes the gpus aren’t available.


What is determining, if the GPUs are available?



 reudrax:

I tried using Bert-large-wwm-cased, it gave this error CUDA error: Device side assert triggered. Immediately after that I ran bert-large-wwm-uncased, it worked fine.


To isolate the assert, you would have to rerun the code with CUDA_LAUNCH_BLOCKING=1 and post the stack trace, which is hopefully different than the previous error.