Solved by ptrblck in post #2 
              
              
                I’m not entirely sure, but you could try to use the unreduced loss via reduction='none', apply your custom operations, and reduce it afterwards.