Solved by malbergo in post #2 
              
              
                This might have been resolved by making two separate “label” variables (for real and fake) rather than reusing. 
I overlooked how this variable was involved with the gradient computation.