Ah, ok thanks! I appreciate the pointer to the other docs I will also check them!
My current goal is to give a chunk of batches or single examples in a list to a model (that is shared in parallel). Then do the optimizer step once all the losses have been computed (like in my example). Will have to read DDP to see how thats done there (note one of my model takes in a list of examples while the other does take a tensor). Will try to parallelize both.
——
I basically want to translate my self contained exampled to the right pytorch parallelization.