i realized my model was way too dense to keep track of gradients, so i converted my torch.cat ops to addition ops
torch.cuda.empty_cache() works, but the problem is it takes up training time, so i called it once after every epoch, and now everything works.
thanks all