Thank you for that tip… Seems crazy to me that that option is enabled by default… That was, of course, the reason that it was being deleted. As for why it’s not working, still trying to figure that out. My model is created like:
from transformers import BertLMHeadModel, Trainer, TrainingArguments, BertTokenizer
model = BertLMHeadModel.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', truncation=True, padding=True)

Later, my dataset is created like this:
dataset = Dataset.from_pandas(pd.DataFrame(model_inputs, columns=['text']))

def tokenize_function(examples):
    return tokenizer(examples["text"], padding='max_length', max_length=90, add_special_tokens=True)

def make_labels(examples):
    result = {k: v for k, v in examples.items()}
    result['labels'] = copy.copy(result['text'])
    return result

labeled_dataset = dataset.map(make_labels, batched=True)
tokenized_datasets = labeled_dataset.map(tokenize_function, batched=True, num_proc=64, remove_columns=["text"])