I guess “crashes the RAM” means that you are running out of host memory?
If so, did you try to delete the previous LightningModule as it seems to load the entire dataset in its __init__ method, so it could use more memory compared to using the Dataset and DataLoader outside of this class.