Currently testing out HuggingFace Accelerate on TPUs on Colab based on the example notebook 1.
My question: Do you have to instantiate the dataloaders inside the training function? Because I didn’t, and then I get this error:
UnboundLocalError: local variable 'train_dataloader' referenced before assignment
Exception in device=TPU:5: local variable 'train_dataloader' referenced before assignment
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 329, in _mp_start_fn
    _start_fn(index, pf_cfg, fn, args)
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 323, in _start_fn
    fn(gindex, *args)
  File "/usr/local/lib/python3.7/dist-packages/accelerate/utils.py", line 274, in __call__
    self.launcher(*args)
  File "<ipython-input-20-001611a4ea01>", line 33, in training_function
    model, optimizer, train_dataloader, val_dataloader, test_dataloader
UnboundLocalError: local variable 'train_dataloader' referenced before assignment
Exception in device=TPU:4: local variable 'train_dataloader' referenced before assignment

It’s weird, because accelerator.prepare is defined within the training_function. The train_dataloader is a global variable, is that the problem?
cc @sgugger