Hi, in case you still have this problem, I think this could work, it may not be the most optimal solution, because you almost make 2 forward passes and manually toggle the requires grad attribute, which I do not know the computational cost so beware of that.
import torch
import torch.nn as nn

# Example layers
fc1 = nn.Linear(1, 1)
fc2 = nn.Linear(1, 1)
fc3 = nn.Linear(1, 1)
fc4 = nn.Linear(1, 1)
wrapper = nn.Sequential(fc1, fc2, fc3, fc4) # To make zero_grad

# To make the forward of a module, without storing the grad
# for itself, but also without cutting the computation graph.
def pass_no_grad(input, module):
    prev_status = {}
    for name, param in module.named_parameters():
        prev_status[name] = param.requires_grad
        param.requires_grad = False
    out = module(input)
    for name, param in module.named_parameters():
        param.requires_grad = prev_status[name]
    
    return out

t = torch.randn(1, 1)
wrapper.zero_grad()

x1 = t.clone()

# First path
x1 = fc1(x1)
x2 = x1.detach().clone() # Cut the gradient of the other path
x1 = pass_no_grad(x1, fc2)
x1 = pass_no_grad(x1, fc3)
x1 = fc4(x1)
x1.backward()
print('First path grads')
print('1', fc1.weight.grad)
print('2', fc2.weight.grad)
print('3', fc3.weight.grad)
print('4', fc4.weight.grad)

wrapper.zero_grad()

# Second path
x2 = fc2(x2)
x2 = fc3(x2)

x2.backward()
print('Second path grads')
print('1', fc1.weight.grad)
print('2', fc2.weight.grad)
print('3', fc3.weight.grad)
print('4', fc4.weight.grad)

'''Output
First path grads
1 tensor([[-0.1179]])
2 None
3 None
4 tensor([[-0.8320]])
Second path grads
1 tensor([[0.]])
2 tensor([[-0.8599]])
3 tensor([[0.0585]])
4 tensor([[0.]])

'''


As you can see, on one path only the fc1 and fc4 layers have a gradient. In the other path only fc2 and fc3 have non-zero grad, but since the other modules still have requires_grad=True, their gradient is not None, but because it is zero it won’t modify the weight’s values on the optimizer step.
Please let me know if this is what you needed.