I world like to L2 toward initial value for embedding model.
code snipe in Theano ([link]
(https://github.com/jwieting/iclr2016/blob/master/sentiment/lstm_model_sentiment.py 2))
    l2 = 0.5*params.LC*sum(lasagne.regularization.l2(x) for x in self.network_params)
    if params.updatewords:
         return l2 + 0.5*params.LW*lasagne.regularization.l2(We-initial_We)
     else:
         return l2

In paper 7, the author said “All models use L2 regularization on all parameters, except for the word embeddings, which are regularized back to their initial values with an L2 penalty”.
But I don’t know how to “regularized back to their initial values”.
I have tried this. But it did not work as expected.
    optimizer = optim.Adam([
            {'params': model.parameters(), 'lr':args.lr, 'weight_decay':args.wd },
            {'params': embedding_model.parameters(), 'lr': args.emblr, 'weight_decay':args.embwd}
        ])