Is there a way to batch examples by max number of tokens using tokenizers and datasets?
I want to batch my dataset ("super_glue") by the number of tokens instead of the number of sentences. Does this library have any Sampler or Collater that can do this while using a huggingface tokenizer and a huggingface dataset?
Yes, see this answer on our forum: https://discuss.huggingface.co/t/are-dynamic-padding-and-smart-batching-in-the-library/10404/4<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Sorry for bumping this up -- I had the same question but I think the page linked here is answering a different question.

Let me ask again differently: is there a [fairseq `--max-tokens` counterpart](https://github.com/facebookresearch/fairseq/issues/165) I can use in huggingface?