Could you rerun your code with torch.backends.cudnn.benchmark = True and check, if you are seeing a speedup?
This setting will profile different cudnn kernls for your current workload and will select the fastest one.
Also, could you post the definition of your dilated convolutions, please?