Hi,
I depends on what you want.
Let’s put an example

Here the error wrt i1 depends (at some point given chain rule) of w1 and w3, which depends on h1 and h2. So if you don’t track gradients it is equivalent to delete the edge which goes from h1 to i1.
Is it convinient? Depends on your model.