The network architecture that I am using is this one: https://github.com/bluestyle97/STGAN-pytorch 8
It is a port of STGAN to PyTorch.
When training on one GPU, I use a batch size of 32. When training on 2 GPUs, I use a batch size of 64, as to still keep 32 samples on each GPU.
I know that the speedup will not be by a factor of 2, however, the network is slower by a factor of 4, compared to running it on a single GPU.