So I guess the answer is probably not.
Im not sure about splitting layers because the backprop still would have to go thru those neurons, only they would simply let the gradients pass unmodified; I would have to see how to rewire the graph. Even if that were possible, Im trying to do something dynamic - a bit like dropout but a. not random but assigned b. on gradients c. i dont want to kill the gradients that go there, just let them thru.  I wonder if the overhead of rebuilding the layers is not more than flattening the gradients before backprop.
In my case ideally one would pass a mask to the forward graph that only computes gradients on some layers assigning a passthru to the others. But I dont know enough about the guts of pytorch or cuda to understand if that would break the parallel computations.
I guess I just will have to test to measure how costly this is.
thanks