Let me break down the problem.
I have a network of say 5 convolutional layers, wherein only 4th layer is a custom convolution layer as you saw in above code.
The output shape from forward of this 4th i.e. custom layer is ([[batch_size,128,4,4]).
Now, I have to include an additional loss (apart from cross entropy at the end of network which is usual in image classification task) in the backward computation of the layer itself, which takes in features (Zi, Zj, Zk) from that output to let the network learn them.
Sorry, there is one typo. Correction : cont_loss = torch.tensor([0.]).to(dev) and cont_loss = cont_loss.add(Li)
cont_loss is that additional loss that you have noticed in the code. For including this only I have to manipulate the backward so that everything works as usual when I call loss.backward() in training loop.
That’s why I have these two additional terms which need to be taken care :
grad_weight += cont_loss_weight and grad_bias += cont_loss_bias.
But, I don’t know which of my idea is correct that I wrote before.
I hope I could elaborate.