Solved by ZimoNitrome in post #3 
              
              
                Apparently "it just works"™ 
The key might be to backprop in different stages. More specifically the following training scheme seems to work well: 


Train D on real images 


Train D on generated images 


Train G with loss from D 


Train G with any other objectives (i.e. L1) 


After each stage, …