Hi Thomas,
so you are using AMP with gradient scaling?
Gradient hooks will slow things down a bit, but maybe not terribly.
One quick thing you could try is whether the problem happens with a particular (group of) parameter(s) and use a larger eps only for them (by using parameter groups).
Best regards
Thomas