The patch is probably mostly 0, 1, after clamping, right?
If that is the case then the gradient is 0, as changing the the clamped value a little bit would not change the output. This is why many nets use squashing to 0…1, e.g. with sigmoid, instead, to move their outputs into the desired range.
Best regards
Thomas