You would need to train these layers, otherwise you should get random outputs.
Is the original paper also returning the output in the original shape?
If so, could you explain their architecture a bit or link the paper here?