The parameters might still get updated even with a zero gradient, if you are using an optimizer with momentum or other running estimates.
Here is a small code example:
model = nn.Linear(1, 1, bias=False)

optimizer = torch.optim.SGD(model.parameters(), lr=1., momentum=0.) # same results for w1 and w2
#optimizer = torch.optim.SGD(model.parameters(), lr=1., momentum=0.5) # w2 gets updated
#optimizer = torch.optim.Adam(model.parameters(), lr=1.) # w2 gets updated

w0 = model.weight.clone()

out = model(torch.randn(1, 1))
out.mean().backward()
optimizer.step()
w1 = model.weight.clone()

optimizer.zero_grad()
optimizer.step()
w2 = model.weight.clone()

print(w1 - w0)
print(w2 - w1)

You can comment out/in the different optimizers and will see that w2 will differ in the second and third use case.
For your use case I think using different optimizers might work best, such that you would only call optimizer_taskX.step() using the corresponding optimizer.