So to summarize, I can just use the following snippet in my script and it will automatically know to skip those dataset processing lines? I assume that it will load in the cached dataset in the Dataset.from_dict and then skips all operations on the dataset by checking whether they are already done in the hash. I do not know enough about hashing to know how this exactly works. If I tokenize with another tokenizer, will it not use the cached version and do the tokenization from scratch with the new tokenizer?
That being said, that does mean that the tokenizer is created but not used (at this stage) as is keys_to_retain . That is not a problem; I am just trying to understand.
def create_dataset(data_path):
    dataset = Dataset.from_dict(torch.load(data_path))
    tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
    keys_to_retain = {"input_ids", "sembedding"}
    dataset = dataset.map(lambda example: tokenizer(example["text"], padding='max_length'), batched=True)
    dataset.remove_columns_(set(dataset.column_names) - keys_to_retain)
    dataset.set_format(type="torch", columns=["input_ids", "sembedding"])
    return dataset

ds = create_dataset(data_p)
dataloader = Dataloader(ds, ...)