gc.collect() shouldn’t be necessary to avoid running out of memory.
Could you check your data loading functions, if unnecessary data samples are stored somehow?
Are you saving the model parameters somewhere, e.g. to restore the model for the best epoch?