How to use multi-node multi-gpu cluster for large batch beam-search decoding
I have a self-implemented-from-scratch torch.nn.Module GPT2 model (self-implemented for custom tweaks to the model) and now I have a 4-node-32-GPU cluster which I like to use to beam-search generate continuation for millions of prefixes. Is there a way to use hugginface as a wrapper to do this in a multi-node-multi-gpu fashion? I know it can be done by manually splitting the file into small chunks and individually beam-search generate,but it is messy.
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.