text: Title: Ray tests are flaky
  Body: Ray tests have shown to be flaky, especially with GPU (Buildkite CI).
  
  There are two places that cause these flakiness:
  1. Some tests fetch `ray.available_resources()` at the beginning of the test and compare the `dict` against `ray.available_resources()` after the test: `assert check_resources(original_resources)`. It looks like that `dict` has race conditions: https://buildkite.com/horovod/horovod/builds/7306#cc0600a5-13ed-479d-ba1b-3bb0f6847992/332-436
  ```
         AssertionError: assert {'CPU': 4.0, ...147712.0, ...} == {'object_stor... 9999999984.0}
           Differing items:
           {'object_store_memory': 10000000000.0} != {'object_store_memory': 9999999984.0}
           Left contains 5 more items:
           {'CPU': 4.0,
            'GPU': 4.0,
            'accelerator_type:T4': 1.0,
            'memory': 189132147712.0,...
  ```
  2. Some tests see more GPUs than there should be:
  ```
         assert len(all_envs[0]["CUDA_VISIBLE_DEVICES"].split(",")) == 4
         assert 8 == 4
           +8
           -4
  ```
  First, the tests should work with any number of GPUs (larger than the minimum required GPUs of 4). Second, the test run on a machine with only 4 GPUs on Buildkite, with only one agent per machine. So there cannot be more than 4 GPUs visible to those tests. It looks like the `RayExecutor` provides that environment variable to the worker and somehow start 8 workers rather than 4.