Will data be loaded into multiple GPUs automatically?
Will models that use TensorFlow as a backend be loaded into all available GPU memory or not?

I'm interested in training the OPT model's 1.3b and 30b variants. However there is no cloud computing GPU i can use which has more than 16gb at a time. The problem is that each individual sample in the batch is at least 24gb, when making use of tensorflow, will the memory be distributed among GPUs or will I still get an allocation error?
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Maybe of interest for @Rocketknight1 @gante<|||||>Hi @Leli1024 👋 By default, TensorFlow will allocate memory in all available GPUs, but will only execute in one GPU -- meaning that what you want to do won't happen by default. At the moment, we have no multiple GPU examples (@Rocketknight1 correct me if I'm wrong), so you will have to build a custom solution for yourself.

Here is a TensorFlow guide for it: https://www.tensorflow.org/guide/distributed_training<|||||>We're co-ordinating with the TensorFlow team to make some examples of exactly this process available using the new `DTensor` API introduced in TensorFlow 2.9. I suspect it'll be another month or so before they're available, though!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.