Sentient07:

I am curious why training on 2 GPUs fail while it progresses well on single GPU?


It might be “bad luck” due to a change in memory fragmentation, as I still think you might be close to the limit:

Tried to allocate 8.51 GiB (GPU 0; 10.73 GiB total capacity; 1.19 GiB already allocated; 8.51 GiB free; 109.16 MiB cached)

so it seems that vert little memory might be missing.



 Sentient07:

Could you please elaborate this?


The docs 1 explain the general use case and this notebook 3 gives you an example (the notebook is quite old by now, but should still give you an idea of the usage).