Hi,
I am happy that you would like to review my code. I created a docker image, and you can see my problem by pulling and running it:
$ docker pull coincheung/debug_zzy:first
$ nvidia-docker run -it --ipc=host coincheung/debug_zzy:first bash

To reproduce the apex training, the command is:
# cd /debug/bisenetv2
# CUDA_VISIBLE_DEVICES=6,7 python -m torch.distributed.launch --nproc_per_node=2 train.py

And in order to switch to the pytorch native fp16, the command is:
# cd /debug/bisenetv2
# CUDA_VISIBLE_DEVICES=6,7 python -m torch.distributed.launch --nproc_per_node=2 train_amp.py

On my platform, the training time of apex is around 85s/100iter, and the pytorch native fp16 traing time is around 115s/100iter.