Ah, I need to do the same while training a model on a very large tree where each node has a coefficient vector and I only need to backprop over the ones traversed in a given run.
Did you see any problems with your approach?