Usually NN models are trained using mini-batches. That’s why I suggest to just drop the mean loss backward part. With your code, you are trying to train with BOTH mini batches (the inner loss backward) AND also dataset/dataloader wise (once every len(dataloader)) (outer total_loss backward). This is not consistent in that you have so much more mini batches than datasets/dataloaders and you end up having those dataset/dataloader updates having little significance.
If you insist to do averaging, perhaps you can do zipping of the datasets, then in each batch, calculate the loss for each of the batch from difference datasets and then averaging the loss and backward.