Apparently "it just works"™
The key might be to backprop in different stages. More specifically the following training scheme seems to work well:


Train D on real images


Train D on generated images


Train G with loss from D


Train G with any other objectives (i.e. L1)


After each stage, I used backwards() on each network while calling optimizer.step after stage 2 and 4.