ptrblck:

Currently it seems you are subtracting a matrix of the shape [42, 242] with 42 ones in the first diagonal from the weight before squaring and summing it.


That is exactly the idea, yes. This regularisation is supposed to keep the weights of the new dense layer close to the identity matrix (in the regularisation matrix) so that with increasing reg. weight the output of the previous dense layer is not transformed much. I realise now that I could have explained the core idea better.
For this you can just assume that the regularisation is correct. How do I regularise with a custom regularisation scheme like this?



 ptrblck:

Did you try to use the standard weight decay?


That doesn’t seem to offer me a way to incorporate the regularisation matrix if I understand it correctly.
More detail:

I pretrain the model with data A: [emb -> LSTM -> attention -> dense -> softmax]
Afterwards I freeze all the ‘old’ layers and add a dense layer after the original dense (output) layer, so now it is [emb -> LSTM -> attention -> dense -> dense -> softmax], the new dense layer has the dimensions of the original output dense layer and the LSTM layer combined: so dense1(42, 42) + lstm(42, 200) = dense2(42, 242)
Now, while finetuning the new dense layer receives the output of the original output dense layer and the output of the lstm, so: dense2([dense1();lstm()]) -> softmax
The weights of this new dense layer are now supposed to be regularised through the formula that I posted (controlled by the reg. weight)
For this to work I need to add this term to the loss (or at least that is what I have done so far, is there another way in PyTorch?):

image551×518 14.4 KB