Even if you don’t have a large dataset but still can’t fit it in a GPU, you can use lazy loading to mitigate this issue.
One thing I didn’t notice properly from your question earlier is this line,

…it processes some batches and runs out of memory…"

This won’t happen unless you have a variable batch size or data points that have different memory sizes.
One plausible reason could be that variables that you might potentially not be using are getting accumulated on GPU over steps/epochs. See if the following helps,
https://pytorch.org/docs/stable/notes/faq.html 8