Nope, the graph is constructed in the same way. Of course the graph depends on the input. I also tried to substitute my input data with torch.rand() tensor and then measure the forward time (results in the pastebin) and also in this case seems that pytorch 1.2 provides way better performances than 1.5.
Any idea on how to further debug this? Thank you! 
EDIT 1:
Investigating a little bit I’ve produced some tracing plot for PyT 1.2 and 1.5… The problem seems to be in batch normalization layer… am I wrong?
PyTorch 1.2 cudnn enabled:
cuda12trace2760×582 86.1 KB
PyTorch 1.5 cudnn enabled:
cuda15trace2954×452 69.5 KB
Edit 2:
Interestingly… by setting ‘torch.backends.cudnn.enabled = False’ performance gets better (half the CUDA time) with PyTorch 1.5, still more time than with PyTorch 1.2 and cudnn enabled.