Hi Shawn,
Yes we support multi-GPU on a single machine.
Check out our examples:
https://github.com/pytorch/examples/tree/master/imagenet
https://github.com/pytorch/examples/tree/master/dcgan
Also check out corresponding documentation:
http://pytorch.org/docs/nn.html#multi-gpu-layers 1.1k