As I said I am already training one dataset (Xi, Yi) at a time, saving the parameters Pi, re-initializing, and doing it again for (Xi+1, Yi+1) and so on.
My question was, since it’s the same model and loss function and optimizer each time, is there an efficient way for me to train on these different datasets at the same time, like I train on multiple examples within one dataset at the same time via mini batches. Except, instead of averaging the gradients like a minibatch, I want to keep multiple sets of parameters Pi, one for each dataset (Xi, Yi), and only update Pi with the gradients associated with the examples in the minibatch that come from (Xi, Yi).