Hi Wei!



 weiguowilliam:

My core question is: can we put all these terms into a list, use a list of optimizer/loss, update list_of_optimizer[i], backward the list_of_loss[i]?


Yes, absolutely this will work.  It’s a perfectly reasonable approach
to take.  Of course, as you saw with the issue in your first version,
you have to do it correctly, but that’s true with any code.


list_of_loss = []
...
for epoch in range(num_epochs):
    list_of_loss = [0 for _ in range(num_model)]

    for j, (images, labels) in enumerate(train_loader):
        for i in range(num_model):
            ...
            list_of_loss[i] = list_of_cri[i](output, labels)
            ...
            list_of_loss[i].backward()



A few comments, mostly stylistic:
Your first list_of_loss = [] has no effect because you subsequently
overwrite it with list_of_loss = [0 for _ in range(num_model)]
before you ever use it.
I would create the list of length num_model once, outside of all of
the loops.  Stylistically, I would initialize the elements of the list with
None rather than 0.  (These initial values are never used, so which
initialization you use has no practical effect.)  I would also not use
a list comprehension merely to create a list of a given length.  Thus,
I would probably do something like this:
list_of_loss = [None] * num_model
only_need_one_cri = nn.CrossEntropyLoss()
...
for epoch in range(num_epochs):
    for j, (images, labels) in enumerate(train_loader):
        for i in range(num_model):
            ...
            list_of_loss[i] = only_need_one_cri (output, labels)
            ...
            list_of_loss[i].backward()

Also, I’ve illustrated that because an instance of CrossEntropyLoss
contains no state (that depends on i, list_of_model[i], or whether
its been called before), we don’t need a list_of_cri that contains
multiple instances of CrossEntropyLoss – a single instance suffices.
Best.
K. Frank