Hello,
It seems you’re using a model structure like concatenating two encoder-decoder type network which each one has a 1 depth. You are probably facing vanishing gradient problem so for being sure about that you can control output of each layer.