I had to follow the training procedures of the paper more closely and base my code on DCGAN. Eventually I made it work, and you can take a look at my repo (it also contains code for swapping latent encodings between images but you can turn it off). the repo is at https://github.com/DimTrigkakis/TrueRelations 137