Hi,
I have a different but related problem.
I have a detection model with unfixed input size. At some extreme case, the RuntimeError with OOM occurs, so I wrap the forward+backward around try+except
for images, targets in data_loader:
        images = images.to(device)
        targets = [target.to(device) for target in targets]
        try:
            loss_dict = model(images, targets)
            losses = sum(loss for loss in loss_dict.values())
            optimizer.zero_grad()
            losses .backward()
        except Exception as ex:
            torch.cuda.ipc_collect()
            torch.cuda.empty_cache()
            continue

this would help to some extend, until:

RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing its output (the return value of forward). You can enable unused parameter detection by passing the keyword argument find_unused_parameters=True to torch.nn.parallel.DistributedDataParallel. If you already have this argument set, then the distributed data parallel module wasn’t able to locate the output tensors in the return value of your module’s forward function. Please include the structure of the return value of forward of your module when reporting this issue (e.g. list, dict, iterable). (prepare_for_backward at /opt/conda/conda-bld/pytorch_1556653099582/work/torch/csrc/distributed/c10d/reducer.cpp:408)

To be clear, I already set find_unused_parameters=True, and broadcast_buffers=False. My guess is some internal state was strained somehow. Is it possible to reset all the state during my except catch?