Your code should be alright.
Basically the computation graph will be built in the same manner regardless if you put your custom layer inside the model or the loss function. As long as the computations stay the same, you’ll get the same result.
Do you encounter any issues with your approach?