The simplest way (but not the most efficient one) would be to create a new variable/tensor with the same data (eg with detach) and then pass it separately through the last layer and apply the loss function afterwards.
A separate optimizer is (from my understanding) not working because the optimizations could lead to different optima and therefore kind of “revert” the steps of the other optimizer