Batching BPE Tokenization Merges
<|reference_start|>Batching BPE Tokenization Merges: The Byte Pair Encoding algorithm can be safely batched to merge hundreds of pairs of tokens at a time when building up a tokenizer's vocabulary. This technique combined with reducing the memory footprint of text used in vocabulary training make it feasible to train a high quality tokenizer on a basic laptop. This paper presents BatchBPE, an open-source pure Python implementation of these concepts, with the goal of making experimenting with new tokenization strategies more accessible especially in compute- and memory-constrained contexts. BatchBPE's usefulness and malleability are demonstrated through the training of several token vocabularies to explore the batch merging process and experiment with preprocessing a stop word list and ignoring the least common text chunks in a dataset. Resultant encoded lengths of texts are used as a basic evaluation metric.<|reference_end|>
@article{morgan2024batching,
    title={Batching BPE Tokenization Merges},
    author={Alexander P. Morgan},
    journal={arXiv preprint arXiv:2408.04653},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.04653},
    primaryClass={cs.CL cs.AI}
}