Solved by ptrblck in post #4 
              
              
                Your use case sounds similar to the pix2pix models so you could check some implementations and reuse the logic for the generator.