my suggestion is to buy new GPU 
but I have another suggestion for you, try to limit your batch size. Your problem is not in your custom dataset I think. The problem happen when batch iteration for validation dataset is loaded into gpu memory, and your gpu capacity does not fit with that…