text: Title: Mixed Precision Training Error, Missing ranks in validation.
  Body: **Environment:**
  Horovod v0.21.0:
  
  Available Frameworks:
      [X] TensorFlow
      [ ] PyTorch
      [ ] MXNet
  
  Available Controllers:
      [X] MPI
      [ ] Gloo
  
  Available Tensor Operations:
      [X] NCCL
      [ ] DDL
      [ ] CCL
      [X] MPI
      [ ] Gloo
  
  TF 2.4 Successfully opened dynamic library libcudart.so.11.0. 2.4.0
  
  
  **Bug report:**
  
  
  During training process if I turned on mixed precision in tf with code 
  ```python
  policy = tf.keras.mixed_precision.experimental.Policy("mixed_float16", loss_scale="dynamic")
  tf.keras.mixed_precision.experimental.set_policy(policy)
  ```
  it will keep return results like, and I am using 2 machines 2gpus/each with batch size 32. Also, it will not throw this error immediately, normally after 10 epochs.
  ```python
  [2022-01-26 05:54:49.745777: W /tmp/pip-install-ypgzgfmp/horovod_f2972a35cff84b9fb159b78871eab48a/horovod/common/stall_inspector.cc:105] One or more tensors were submitted to be reduced, gathered or broadcasted by subset of ranks and are waiting for remainder of ranks for more than 60 seconds. This may indicate that different ranks are trying to submit different tensors or that only subset of ranks is submitting tensors, which will cause deadlock.
  Missing ranks:
  0: [cond_1/then/_400/cond_1/Adam/PartitionedCall/DistributedAdam_Allreduce/cond/then/_2867/DistributedAdam_Allreduce/cond/HorovodAllreduce_DistributedAdam_Allreduce_cond_Cast_1_0, cond_1/then/_400/cond_1/Adam/PartitionedCall/DistributedAdam_Allreduce/cond_1/then/_2875/DistributedAdam_Allreduce/cond_1/HorovodAllreduce_DistributedAdam_Allreduce_cond_1_Cast_1_0, cond_1/then/_400/cond_1/Adam/PartitionedCall/DistributedAdam_Allreduce/cond_10/then/_2947/DistributedAdam_Allreduce/cond_10/HorovodAllreduce_DistributedAdam_Allreduce_cond_10_Cast_1_0, cond_1/then/_400/cond_1/Adam/PartitionedCall/DistributedAdam_Allreduce/cond_100/then/_3667/DistributedAdam_Allreduce/cond_100/HorovodAllreduce_DistributedAdam_Allreduce_cond_100_Cast_1_0, cond_1/then/_400/cond_1/Adam/PartitionedCall/DistributedAdam_Allreduce/cond_101/then/_3675/DistributedAdam_Allreduce/cond_101/HorovodAllreduce_DistributedAdam_Allreduce_cond_101_Cast_1_0, cond_1/then/_400/cond_1/Adam/PartitionedCall/DistributedAdam_Allreduce/cond_102/then/_3683/DistributedAdam_Allreduce/cond_102/HorovodAllreduce_DistributedAdam_Allreduce_cond_102_Cast_1_0 ...]
  1: [HorovodAllreduce]
  2: [HorovodAllreduce]
  3: [HorovodAllreduce]
  ``` 
  
  But If I turn out mixed-precision in tf, it will be training as expected.
  here is my command line code
  ```shell
  #!/bin/bash
  
  mpirun -np 4 -H ${host A}:2, ${host B}:2 \
  --allow-run-as-root -bind-to none -map-by slot -mca \
  plm_rsh_args "-p 20385" -x NCCL_SHM_DISABLE=1 -x NCCL_IB_DISABLE=1 \
  -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH \
  -x NCCL_SOCKET_IFNAME=enp72s0,enp3s0,enp73s0 -mca pml ob1 -mca btl ^openib \
  -mca btl_tcp_if_include 10.0.0.0/16 ./train_test.sh
  
  ```