Ho, right. This is expected then. The cudnn.benchmark takes a very long time the first time it encounters a new input size (as it tests several algorithms).
If your models has many different convolutions with different input sizes, it will do that many times, which can be quite slow indeed.
Happy that you found the reason !