Your approach should work and the computation graph would make sure to calculate the corresponding gradients only.
Could you post your model definition, so that we could check it, if you still have doubts?