Say I want to run a forward pass through the network, using a batch-size of 256, and before backprop I would want to weight the losses across the batch input/target sequences, is that possible and what would be the best practise?
It’s not a solution for me to do preproccessing and collect input/target sequences that will have the same weighted loss/learnrate together and run them in the same batch because what sequence that will be backpropagated with higher learnrate will be determined in comparison with the other sequences in the same batch, after the forward pass.
Is this a catch-22 situation that isn’t possible due to the fact the gradients doesn’t care about the actual different losses inside the same batch? There’s no way to separate different gradients, from different sequences in the same batch, apart from each other, right? Do I have to run the batch once using no_grad and then based on the result filter out/exchange sequences from the batch and then run it again using grad to simulate and approximate this?