Thanks for the suggestion. Tried to implement this, but unfortunately it doesn’t seem to work.
Initialized self.inputs,  self.target and self.g_out to either torch.empty() or torch.zeros(). Tried size of 1 (arbitrarily) and also the true size of those tensors (used as batch size the full batch size divided by the number of GPUs). However, in the second call g_loss = self.train_iter() I get those tensors reset, as if the constructor has been called again (it hasn’t, I added a printout in TrainingIteration.__init__).
To clarify, at the end of forward_discr() I print some value from self.g_out and I print the same value at the beginning of forward_gen(), expecting (or hoping) it would be the same, but it becomes either zero or a random number, depending on whether I initialize self.g_out to torch.zeros() or to torch.empty().
So bottom line, no success so far…
In case there are any additional suggestions as to what can be done, I’d appreciate that. Also, if there is any in-depth material as to how DataParallel works under the hood, that may shed some light on the issue.