I tried removing the scheduler itself from the code and see if it makes any difference but it didn’t.
I hadn’t saved the scheduler when i saved the model, so anyway i can just load optimizer state dictionary and continue training?
EDIT: I had just removed the scheduler.step()
but i had not removed the  following code:
scheduler = lr_scheduler.MultiStepLR(optimizer, milestones=[4, 8, 15, 20, 45], gamma=0.1)

you have to remove this too.
In future its best to save the scheduler dict as @vmirly1 suggested