If the device runs out of memory in the second iteration, it might point towards some intermediate tensors or output tensors, which are still on the device, but could be freed.
E.g. if you are using the training loop as:
for data, target in loader:
    optimizer.zero_grad()
    output = model(data)
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()

the output tensor would still be on the device (as well as the loss tensor, which can be ignored, if it’s just a scalar). You could remove it via del output and rerun the code.
Also, the gradients will be accumulated and zeroed out in the next iteration, which would still use the memory.
If you are close to the OOM, you could remove them by setting all gradients to None via:
for param in model.parameters():
    param.grad = None