Thanks for the reply. I ran into one problem whilst running the distilbert binarization. Can anyone tell me if this means that my data per line exceeds what the distilbert model expects?
!python scripts/binarized_data.py \
--file_path data/dataemail.txt \
--tokenizer_type bert \
--tokenizer_name bert-base-uncased \
--dump_file data/binarized_text

07/19/2020 07:25:56 - WARNING - transformers.tokenization_utils_base -   Token indices sequence length is longer than the specified maximum sequence length for this model (626 > 512). Running this sequence through the model will result in indexing errors