I’m not sure if I understand the question properly, but you would apply the chain rule and thus the conv output would be used.
The general workflow of the chain rule and backpropagation is explained e.g. in CS231n - Optimization 2.