Hi @dejanbatanjac,
You can take a look at this paper Deep Invertible Networks 197 and its (PyTorch) official implementation 229. 
It propose two fully invertible architectures (section 3.1), the first one being only injective and the second one bijective. The two are actually made of two sub-models: a mapping Φ and it’s pseudo-inverse Φ-1, the latter being built with inverse and pseudo-inverse blocks/layers from Φ.
The authors focus classification, so I’m not sure how (good) you could reuse this work for segmentation task