yes!!! @alexeib it worked for me too. I set out_padding = 0 and use F.pad to align the size.  Thanks a lot for the suggestion. But still I will be waiting for some possible explanations before I settle with the solution.