@go-inoue Did you find an answer to this question?
My best guess:
1 000 000 steps equals approx. 40 epochs -> (1*e6)/40=25 000 steps per epoch.
Each step (iteration) is using a batch size of 128 000 tokens -> 25 000 * 128 000= 3.2 billion tokens in each epoch.
One epoch is equal to one full iteration over the training data. In other words the training data contains approx. 3.2 billion tokens.
I would expect the number of tokens to be higher than the number of words in the training data given that full stops, commas etc are separate tokens, and words sometimes are split into several tokens using the BERT-tokenizer.
Could it be that the 3.4 bn word corpus is split into training, validation and test-data? I’m not even sure you have a split of train/val/test data during pre-training of BERT, given it is unsupervised? Some kind of cross-validation of all data would seem to make more sense?