Solved by Oli in post #3 
              
              
                I second @shirui-japina’s suggestions. Do a nn.LossFn(reduction='none'), weigh the different samples to your liking and then do your own reduction (e.g. mean).