If PGGAN refers to “Progressive Growing of GANs” note that the author used 8x 16GB GPUs. If your GPUs have less memory, you won’t be able to train the model completely.
However, are you seeing an increasing memory usage during training? The mentioned step should come pretty at the beginning, so I just would like to make sure you are not storing the computation graphs unnecessarily.
What batch size are you currently using?