This is not really easily readable code… A smaller example that reproduces just this issue would be more helpful
I would check that torch.cat((s_input_var, t_input_var), dim=0) this op actually returns enough sample for the data parallel to work wiht? Or is it explicitly done such that s_* run on one gpu and t_* runs on the other and you always have 2 gpus?