If I understand correctly, you are showing the logs of the training, that you did three times on different datasets (first run aug_p=0.1, second run aug_p=0.2, third run aug_p=0.3) ?
The 2nd and 3rd times it’s loading from the cache the tokenized data. It means that it considers that the dataset you passed (even though aug_p changed) is the same as before.
An easy way to debug this and see if this is the case would be to do
for aug_p, balanced_datasets in synonym_aug_datasets.items():
    print(balanced_datasets["train"]._fingerprint)

If the fingerprints are the same, then it means the datasets are the same.
Could you verify the fingerprints please ? This should be helpful to debug things.
However your function is random, and you said that you got a warning about the cache - it must have notified you that it used random fingerprints because of the complex nature of your function.
Can you confirm this is what the warning message said ?