TL;DR: Torch7 model took 0.5s/batch; same model on PyTorch runs 1.3-6s/batch(x2-3 slower). How should we diagnose the bottleneck?
We have a DenseNet-like model forked from fb.resnet.torch receiving 720x720 images as input. For our original model in Torch7, a forward pass on a 8 TitanXp node took around 0.5s with a batchsize of 112. But after migrating the same model structure to PyTorch, while the backward pass became slightly faster(0.7s to 0.5s) and the max batchsize capacity increased(112 to 120), the forward pass became x2-3 slower(1.4-7s/batch), even with the same batchsize. (By forward pass, I mean the line that goes output = model(x).)
Also some interesting facts:

We initially tried Tensorflow for migration, but its max batchsize was too low for our need.
The default PyTorch implementation of DenseNet in torchvision, while having less layers than our model, is about 10% slower with 50% lower max batchsize.
GPU utilization falls more frequently for PyTorch than Torch7; while Torch7 model reaches 80C+ temperature, PyTorch model stays at 60-70C.
A PyTorch single GPU run with 16 images per batch gives 0.2s/batch, which is just the original speed divided by GPU count(seemingly  no parallelized gains of using 8 GPUs).

Some methods we tried for pytorch:

Using Docker to control the experiment environment, except CUDNN version(5 for Torch7, 6 for PyTorch)
Attempts at optimizing PyTorch code according to the docs: namely, using nn.DataParallel, nn.Sequential, inplace=Trues, pinned_memory and async, making sure nccl is working, adjusting num_workers, mimicking structure from torchvision examples, turning cudnn.benchmark on, not DataParallelling the last fully-connected layers.

I understand that our input size is unusually large, and that PyTorch is still in beta phase; and also, since we cannot disclose our model structure, the question must generalize: Are there pitfalls to avoid and hacks to try in optimizing model speed?