If you could provide executable code which shows what you want, then it would be easier to figure out the issue. If you could make the code as simple as you can while still illustrating the question, that would help, too.
In the example code you could, for instance, have the math operation sandwiched between just two generator layers (I am not sure what these are, from your description …), if such a setup will also illustrate the issue.