did some timing.
using multi-gpus makes a type of loss acts up.
the loss runs on cpu.
so, not sure why is that.
when using a single gpu, the loss takes 70ms to forward.
when using multigpus, it takes 600ms.
looking why this is happening.
the observed gain in time of 15% came from the validation that is done in multigpu.
thanks