I must exclude all L2 parameters from optimizer

That is sufficient. But from a memory/time standpoint, the recommended practice would be to set requires_grad=False as well. Remember, you’re only freezing the weights (parameters) of L2, not the intermediate computations in that network. When backprop does its thing, the gradients of the loss function WRT the weights (parameters) of L2 are never required for updating the weights of L1 or L3.
I think you and I had the same misunderstanding of requires_grad=False. It helped me a lot to watch Elliot’s video and visualize the intermediate layer (i.e. L2) not as a lumped network, but as a connection of parameters being fed to computations.