This warning should indicate that some of the calculated gradients are non-finite (Inf or NaN most likely). I would claim it depends on your use case, if these invalid gradients are expected and if clipping them should be fine or if you would like to avoid them in the first place.