I had one another doubt. I wanted to add forbenius norm of covariances with my cross entropy.
i have tried this:
class f_norm(nn.Module):
    def __init__(self,epsilon,beta):
        super(f_norm,self).__init__()
        self.epsilon = epsilon
        self.beta = beta
    def forward(self, cs,ct):
        epsilon=0.1
        beta = 0.2
        x = self.epsilon*( torch.sqrt((torch.abs(cs-ct))**2)-self.beta)
        y = x.view(-1)
        y = torch.sum(y)
        return y/1000

This doesn’t seem to work, it doesn’t return a Tensor.
I am using this function instead:
def f_norm(cs,ct):
    epsilon=0.1
    beta = 0.2
    x = epsilon*( torch.sqrt((torch.abs(cs-ct))**2)-beta)
    y = x.view(-1)
    y = torch.sum(y)
    return y/1000

loss = cross_entropy(inputs,targets)+f_norm(cs,ct)

If I call loss.backward(), would this work fine ? I read in forums that unless it’s autograd variable it would not be updated. So was having concerns with this implementation.