Model parallelism support?
Hi. I'm running into memory issues when trying to fine tune the CTRL language model on a 16 GB GPU. Is there any built-in support for splitting models across more than one GPU? I suppose that mapping the input embedding layer to a different GPU or even the CPU would do the trick.

How do you training CTRL ? Did you check this: https://github.com/huggingface/transformers/blob/0477b307c7501ea76e01b03cb387a2312db752b3/examples/run_lm_finetuning.py#L198

You can also decrease batch size up to 1.

And also gradients can be accumulated:
https://github.com/huggingface/transformers/blob/0477b307c7501ea76e01b03cb387a2312db752b3/examples/run_lm_finetuning.py#L389<|||||>@iedmrc Thanks for the suggestions. I'm using a slightly modified version of run_lm_finetuning.py but for CTRL even with a batch size of 1 the fine tuning consumes more than 16 GB of memory. I tried also to fine tune with half precision (fp16), but for some reason that didn't help either (I have seen other people complain about similar issues here).<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>See https://github.com/huggingface/transformers/pull/3578. 