Hello everyone! I have a simple problem: how can I constrain the weights of a convolutional layer to have unit norm?
I currently have the following working example code
filter = torch.tensor( np.array([1,1,1,1]), dtype=torch.float, requires_grad=True)
filtered_out = torch.nn.functional.conv1d(input, filter , stride=1)

The way I was thinking to normalize the weights was
filter = torch.tensor( np.array([1,1,1,1]), dtype=torch.float, requires_grad=True)
filter_norm = filter/torch.norm(filter)
filtered_out = torch.nn.functional.conv1d(input, filter_norm , stride=1)

However, I’m a bit afraid that the thing can become unstable (what if the filter norm gets really close to zero?).
The other method I was thinking to use is to reproject the filter tensor to the unit norm sphere after every .backeard() step, that is I divide it by its norm at every step. However, I’m not sure about how to manually change the values of a tensor without breaking the autograd.
Any idea?