If you want to continue the training in apex, then you would have to restore the apex state_dict.
However, we recommend to check out the native mixed-precision implementation as described here 2, which provides a smoother experience than apex.
Note that you would have to install the nightly binaries or build from source to use it.