Ah yes sorry, this code is in a register backward hook function
layer.register_backward_hook(saveAverageD)
...
def saveAverageD(self, grad_input, grad_output):

Sigmoid prime is the derivative of the sigmoid function to calculate the gradient during the backward pass if autograd wasn’t doing it already under the hood.
The module just has a conv2D or Linear layer and then a sigmoid for the earlier layers and it seems to be doing what I want it to do.  But at the top layer where it is just a Linear into the loss function, I do not do this step and just use grad_output it does not appear to be working.