The think is that is you change the number of hidden units in your network, you need to change the layer before and after so that they handle this new size. And thus their weight matrices will have to change. For the rest of the graph, nothing changes and all derivatives will be computed the same.
That being said, when changing hidden layers, you change your optimization objective. So if you change it too often, it may be complex for your optimization to converge properly as the point where it tries to converge to moves every time you change the structure of your model.