I had a similar problem, you might be getting nan's because some of the inputs to the second gamma correction layer are very close to 0 and you raise them to a power which leads to problems with gradients. If I am not mistaken about how it works in PyTorch (roughly), the derivative of x^a is being evaluated with respect to a and with respect to x, either one could cause problems (say, a<1).
Now, I hope this small bit info helps you, because in my case I was unable to resolve my issue.