I did this and the code now runs:
 self.injected_noise = nn.Parameter(self.build_noise_injection(shape))

I set it has a parameter since it is composed of weights * self.noise, where weights is trainable.
So to clarify - do I need to push every single variable in the graph to cuda, either defining it as a parameter or as a buffer?
Much thanks