case1:
input = ...
model1 = ...
model2 = ...

s1 = torch.cuda.Stream()
s2 = torch.cuda.Stream()

with torch.cuda.stream(s1):
    output1 = model1(input)
    optimizer1.zero_grad()
    loss1(output1, label).backward()
    optimizer1.step()

with torch.cuda.stream(s2):
    output2 = model2(input)
    optimizer1.zero_grad()
    loss2(output1, label).backward()
    optimizer2.step()

case2:
input = ...
model1 = ...
model2 = ...

s1 = torch.cuda.Stream()
s2 = torch.cuda.Stream()

with torch.cuda.stream(s1):
    output1 = model1(input)
    optimizer1.zero_grad()
    loss1(output1, label).backward()
    optimizer1.step()

    output2 = model2(input)
    optimizer1.zero_grad()
    loss2(output1, label).backward()
    optimizer2.step()
    

I expected the first case to take about half as long as the second, but I found that both cases took about the same amount of time.
I don’t know why.
How do I implement parallel execution of multiple models in multiple cuda streams？