Avoid flaky trainer test failures
# What does this PR do?

Avoid flaky trainer test failures on multi GPUs.

I ran the test 1000 times on single/muti GPU VMs.

- single gpu: all 0.0
- multi gpu: a lot of 0.0, but the largest can go 1e-4.
_The documentation is not available anymore as the PR was closed or merged._<|||||>> I assume that 1e-7 and 1e-6 were still flakey, right? Just concerned with very low margin where it could potentially hide issues.
> 
> But otherwise looks good. Thank you for fixing, @ydshieh

From Slack report, I could see a few times with `2e-6`. On my manual tests (1000 times), multi GPUs could reach `1e-4`. I choose `1e-5`, thinking the test failure would be super rare.<|||||>Super, thank you for sharing the details of why this value was chosen, @ydshieh!

All is good.