Sometimes I found the restarting you computer works.
It may have something in the memory or holding on to something that you dont know.
If it still keeps doing it, make a data loader and do mini-batches.
That may help