Measuring time when using xla_spawn on multiple cores 
Hi,
I would like to measure the time of the program, when this is launched in distributed way on mutiple TPU cores with xla_spawn, I tried to write some callback like this which retunrs the same results with num_cores = 1 and 8, I am not sure how to handle it when this is distributed? I want to make sure pytorch XLA is using all cores indeed, thank you for your help. 

```
class PrinterTimeCallback(TrainerCallback):
  
    #A bare :class:`~transformers.TrainerCallback` that just prints the logs.
   
    def on_epoch_begin(self, args, state, control, **kwargs):
        """
        Event called at the beginning of an epoch.
        """
        if state.is_local_process_zero:
            self.start_time = time.time()


    def on_epoch_end(self, args, state, control, **kwargs):
        """
        Event called at the end of an epoch.
        """
        if state.is_local_process_zero:
            self.end_time = time.time()
            total_time = self.end_time - self.start_time
            print("total_time ", total_time)

```
   

Hi, I think you should take a look at the [troubleshooting section over at pytorch/xla](https://github.com/pytorch/xla/blob/master/TROUBLESHOOTING.md), it has a lot of information on how to debug on TPU.

The metrics report they mention in the first paragraph can be output by the trainer with the `--debug` flag.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
