PyTorch should be able to reuse the memory and I’m unsure if avoiding the allocation would save you from the downstream OOM.
Anyway, the large block might be caused by testing different cudnn kernels and allocating a workspace for them, if you are using torch.backends.cudnn.benchmark = True. Could you disable it and check, if the memory allocation is reduced? Also cudnn.deterministic = True might help as it shouldn’t use any workspace, if I’m not mistaken.