Supercharge your Pytorch training loop with Hugging Face Accelerate. There are multiple setups on which you can run your training: it could be on CPU, GPUs, TPUs. Distributed on one machine with several devices, or several machines (often called nodes) each with multiple devices. On top of that there are new tweaks to make your training faster or more memory efficient, like mixed precision and DeepSpeed. Each of those setups or training tweaks, requires you to change the code of your training loop in one way or another and to learn a new API. All those setups are handled by the Trainer API, and there are several third-party libraries that can also help you with that. The problem with those is that they can feel like a black box and that it might not be easy to implement the tweak to the training loop you need. Accelerate has been designed specifically to let you retain full control over your training loop and be as non-intrusive as possible. With just four lines to add to your training loop (here shown on the code of the training loop from the "Raw training loop" video), Accelerate will handle all the setups and training tweaks mentioned on the first slide. It's only one API to learn and master instead of ten different ones. More specifically, you have to import and instantiate an accelerator object, that will handle all the necessary code for your  specific setup. Then you have to send it the model, optimizer and dataloaders you are using in the prepare method, which is the main method to remember. Accelerate handles device placement, so you don't need to put your batch on the specific device you are using. Finally, you have to replace the loss.backward line by accelerate.backward(loss), and that's all you need! Accelerate also handles distributed evaluation. You can still use a classic evaluation loop such as the one we saw in the "Raw training loop" video, in which case all processes will each perform the full evaluation. To use a distributed evaluation, you just have to adapt your evaluation loop like this: pass along the evaluation dataloader to the accelerator.prepare method, like for training. Then you can dismiss the line that places the batch on the proper device, and just before passing your predictions and labels to your metric, use accelerator.gather to gather together the predictions and labels from each process. A distributed training script has to be launched several times on different processes (for instance one per GPU you are using). You can use the PyTorch tools if you are familiar with them, but Accelerate also provides an easy API to configure your setup and launch your training script. In a terminal, run accelerate config and answer the small questionnaire to generate a configuration file with all the relevant information, then you can just run accelerate launch, followed by the path to your training script. In a notebook, you can use the notebook_launcher function to launch your training function.