It depends.  Vanilla SGD doesn’t use previous states, so there’d be no point recovering optimizer info for that.   I’d say, if restarting the optimizer isn’t having an averse effect (e.g. you’re not noticing a giant jump when you restart), then you can get by without worrying about it.