Solved by JuanFMontesinos in post #2 
              
              
                There are many reasons. 

The amount of filters of the decoder is bigger.
The kernel size of the decoder is bigger.
Probable the transposed convolution itself is slower. I guess this as you need to allocate more memory (you are upsampling) and to work with sparse kernels.