I am actually facing a problem on my project.
The learning process (using CUDA) is randomly crashing without any error message (most of the time). It can happen after 1min or after 35min.
It only happens if I use both my GPU’s (mismatched 1080Ti and 2080Ti).
If I use only one everything is fine.
We had a power problem (insufficient power supply) but now it is solved and the problem is still here.
Please note that rarely a message about memory access appears (during the crash), not sure it is related. It says an instruction referenced memory that could not be written, always memory with very small address (0x00000…00020)or very big (0xFFFFFFFF…FF).
We also observed that if the model is more heavy the crash happens sooner.
What could cause such crashes  ?
Thanks for your help