In the paper, It’s mentioned that the masing is done on 0.15 f all WordPiece tokens in each sequence at random.
In the code though, it’s based on the inputs in
labels = inputs.clone()
probability_matrix = torch.full(labels.shape, self.mlm_probability)
masked_indices = torch.bernoulli(probability_matrix).bool()

I don’t know whether inputs refers to one sequence or one batch of sequences