What do you mean by “use the gradient of loss with respect to weights of L2 also”? Do the weights of L2 have the same shape of L1 (and do you want to use the gradients of the weights of L2 instead of the gradients of the weights of L1 to optimize)?