I haven’t. That s a good point I will do it later. I realized cuda(non_blocking=True) is not really non-blocking. There is a short delay before it returns. Do you have any idea how I can hide the latency of the cuda() calls? Also what is the idiosyncratic way to train multiple models on different gpus using the same input?
Many thanks!