Actually a convolution is very very sparse.
In the setting where you consider the input as 1D and output as 1D where you just collapse all the dimensions, The Jacobian would be huge (or even writing the convolution as a mm with 1D inputs).
But the weights are both re-used and applied only to subsets of the channels. Leading to a significant sparsity in that operation.