Run a single wandb instance per TPU run
Why is it specific to tpu only?
Would the same logic apply in all cases?<|||||>Yes I'm guessing we only want the global master to log to wandb in DDP, like we do for Tensorboard.

Not sure why we hadn't done it like that before, @borisdayma – thoughts?<|||||>I did not consider DP/DDP at the time. Actually Tensorboard logging does not consider world master either (only for logging config parameters but not metrics).

I understand we should wrap the entire `wandb` and Tensorboard logics within a simple `if self.is_world_master`.

May I suggest the following:

* refactor logging through PR #4756 
* add an equivalent `TFTrainer.is_world_master`
* wrap relevant Tensorboard & wandb sections of `log_metrics` by checking `is_world_master`
* call `setup_wandb` only for world master (checked either within `Trainer` & `TFTrainer` or within `setup_wandb`)

Let me know if you want me to add those changes.<|||||>I think those changes would be welcome. Do you agree @julien-c ?<|||||>Yes I agree. Should we already merge this PR though?<|||||>Sure, it won't do any harm.