Hi @ezyang,
I  solved that  by deleting manually my variables after batch iterations.
Another common solution is :
try:
            try:
                y = net.forward(train_x)
            except RuntimeError as e:
                if 'out of memory' in str(e):
                    print('| WARNING: ran out of memory, retrying batch',sys.stdout)
                    sys.stdout.flush()
                    for p in net.parameters():
                        if p.grad is not None:
                            del p.grad  # free some memory
                    torch.cuda.empty_cache()
                    y= net.forward(train_x)
                else:
                    raise e

Or  “Preview (Nightly)” PyTorch Build