All the 8 GPUs should share exactly the same weights of model (under DDP mode)

Yes they do but if you use these as embedding, each GPU use only a subset depending on the particular input it gets, so you might read some bad values on a GPU and not on another.
fp16 is not incompatible with lr scheduler. but it can lead to instability in quite a few use cases.
That why we actually have the AMP package which is quite complex that is tuned to only use fp16  when it is stable enough.