Maybe @ptrblck may help you in a better way but I think what  you can do is building up the whole model as a single nn.Linear layer and  to set gradients equal to 0 for the corresponding neurons in the backward pass (obviously initializing that weight as zero too)
It’s a little bit mess because weights are passed as a package, so you cannot disable requires grad in a subgraph.
Another option could be to initialize corresponding weights to 0 and  detaching specific neurons during the forward pass.
As you can see it’s a mess.