As far as I know, there is no way to do this automatically. Maybe, you might perform K Nearest Neighbor in latent space to get the nearest image from your dataset.


After the Encoder is trained, you could use the latent vectors to train the Decoder separately without affecting the Encoder (using torch.no_grad() or .detach() etc). This might help to reconstruct the new images (approximately).