Hi,
Gradients give you the direction of steepest ascent. So if you are looking for a descent direction, yes you should flip the signs.
As you saw, in SGD, you do x = x - lr * grad (with a negative sign).