So I think there are at least two differences between the networks at the moment:

You never set the PyTorch model to eval() mode, so you pay additional cost for the Dropout layers (they’re no-ops at eval time, but not at training time).
We’re not using cuDNN for MaxPooling.

I only quickly glanced over the scripts so there might be more.
Benchmark mode doesn’t limit the input size in any way, but it should be used only if you’ll be using a (small) number of input sizes. The benchmarks will be run for every different shape, so if your input wildly varies you might be running them at each iteration. If you train the FCN on a pre-processed dataset where all images are of the same size, use the benchmark mode. If every image is of different size, don’t use it.