process stuck at LineByLineTextDataset. training not starting
# ❓ Questions & Help

I am using below python code.
BASE_MODEL  = "/data/nlp/bert-base-uncased"
CACHE_DIR  = "/data/nlp/cache"

model       = AutoModelWithLMHead.from_pretrained(BASE_MODEL,cache_dir=CACHE_DIR)
t_tokenizer = ByteLevelBPETokenizer()
t_tokenizer.train(files=paths, vocab_size=52_000, min_frequency=2, special_tokens=[
    "<s>",
    "<pad>",
    "</s>",
    "<unk>",
    "<mask>",
])

new_vocab = t_tokenizer.get_vocab()

tokenizer = BertTokenizerFast.from_pretrained(BASE_MODEL)
num_added_toks = tokenizer.add_tokens(list(new_vocab.keys()))
model.resize_token_embeddings(len(tokenizer))
dataset = LineByLineTextDataset(tokenizer=tokenizer,file_path=DATA_FILE,block_size=64)


it works fine for small files but for file > 600 MB the process gets stuck at dataset = LineByLineTextDataset(tokenizer=tokenizer,file_path=DATA_FILE,block_size=64)

Hi, this is because it's tokenizing the entire dataset in a single thread, so it's bound to be a bit slow. This class caches the result though, so you will only have to do this step once!