The error shows that your are running out of memory, so you would need to either reduce the batch size further or try to save memory by e.g. lowering the memory footprint in the overall training using smaller models or by trading compute for memory via torch.utils.checkpoint.