Okay, I have modules module the loss for the main task and the absolute log-determinant for the transformation (for example, the sum of the absolute logs of the diagonal elements for multiplication with a triangular matrix. For non-linear transformation the whole thing is more difficult). Similar to normalizing flows, if you know this approach.
In my earlier layers, I don’t want the weights to optimize the absolute log-determinant because of numerical problems, I only want to avoid “low” log-determinants (< zero). But I don’t know how many layers to disable etc. or whether this would really solve my problem. So I have to experiment a lot.
All the log-determinants of the modules get combined via a sum and added to the overall loss. Since I have not started the problem with the approach in mind, I constantly combine the log-determinants to get better code-organization.