How did you pretrain the Autoencoder?
Based on your explanation, it seems you’ve used the real images:



 che85:

AutoEncoder that was trained to learn the most prominent properties of the structure to segment.


Would this approach work, if you now feed the segmentation output of your VNET to this AE?
The image statistics, ranges etc. should be quite different, so I’m not sure if it’ll work out of the box or if you would need to train both models end-to-end.
It’s an interesting approach and reminds me of  Adversarial Learning for Semi-Supervised Semantic Segmentation 1, but with an AE instead of Discriminator.