I get that you are not “at liberty to disclose the model” but a minimal code snippet does not disclose anything as it is usually quite standard. It might even help you pinpoint the source of the issue by trying to remove anything unnecessary or application-specific. So first step: try to reproduce the error with a minimal script (outside of a jupyter notebook as you might miss a hidden-state).
With that being said, there are a few things you wanna check:

calling eval() after the load_state_dict: you are saying you call eval() during the training ? You should be calling train() in training and eval() when evaluating ?
Are you doing the final test on a different set?
You talk about backbone: when saving the state_dict, are you saving only the backbone’s state_dict or the backbone and the last layer/module (which I’m guessing there is as you are mentioning a backbone). This would explain that it works when saving the whole model.
Try using the cudnn.deterministic flag for evaluation at least

Hope this helps!