Why do we regularize large gradients corresponding to large errors? While reviewing some scientific blogs, I found them recommending using gradient clipping for large error gradients. However, intuitively one would think that when model predictions are completely off, it would make sense to drastically update the weights. Therefore, why do we clip gradients in such cases?
Please note that I am not talking about the exploding gradient problem related to deep architectures. Those problems arise when multiplying gradients across different layers resulting in a large number.

The problem is that a large gradient doesn't necessarily mean that the minimum is far off. Consider e.g. this situation:

where $f$ is the objective function to minimize. Clearly, a smaller step size would have been more appropriate. The learning rate could also be used to regulate this, but that would create only a linear connection between gradient and step size. And with clipping, you have one extra parameter to tune, which gives you more freedom (and more headaches) in adjusting your optimization.
Another cool approach to deal with fast changes in the objective function is the Nesterov accelerated gradient.

You asked why we do clipping when the errors are large. It is difficult to tie the clipping to the size of the error because having a large error doesn't mean that you are far away from a minimum and should make larger steps. This large error might be pretty much the best you can get, because of a bad model or because of a lot of noise.
