This may well be an Apex bug.  About a week ago, for a few days the combination of dynamic loss scaling + Apex DDP was broken in Apex master.  I fixed it in https://github.com/NVIDIA/apex/commit/8437d29505fcc7fad28183395abd89a09a17efe6 13, so maybe a fresh clone + reinstall of Apex will resolve the issue.  Be sure to clean the old install before rebuilding:
pip uninstall apex
cd apex_repo_dir
rm -rf build (if present)
rm -rf apex.egg-info (if present)
git pull
pip install -v --no-cache-dir --global-option="–cpp_ext" --global-option="–cuda_ext" .