fermoren:

Because I´m using more than 2 GPUs, my batch in divided between those two devices for parallelisation purposes, which means than when I access the data in the batch in eval/training, I´m getting just half the batch.


Are you using nn.DataParallel or DistributedDataParallel?
In the former (not recommended) case the batch would be split while passing it to the DataParallel model, so inside the forward of the models you would only see a chunk of the data. The output however is gathered again on the default device and you will see the output for the entire batch.
In the latter case, each process would use a single GPU so would be responsible to load the entire batch.