I’m implementing the max norm constraint as detailed in this post 64. Max Norm would constraint the parameters of a given layer based on the L2 norm of the weights.
It’s my understanding that the operations should be done in-place for memory efficiency. I have the following code.
def forward(self, x):
    x = x.view(-1, n_inputs)
    max_norm_(layer.weight)
    x = layer(x)
return x


def max_norm_(self, w):
    with torch.no_grad():
        norm = w.norm(2, dim=0, keepdim=True)
        desired = torch.clamp(norm, 0, self._max_norm_val)
        w *= (desired / (self._eps + norm))


Is this the correct way to implementing it? Will this interfere with the optimization step? I’m using torch.no_grad() since we don’t need to get gradients for this operation.
Thanks