Hi there,
Note the quotes in the title: that’s not PyTorch which is to blame but the careless user that I am.
Here’s the guilty code:
total_loss = 0.0  # total loss of the epoch

for ...
    loss = criterion(...)
    loss.backward()
    # ...
    total_loss += loss

For reference, here is what’s happening:
total_loss is initially a Python float variable, because we use the += operator on it, it is transformed into a Tensor. Because loss has requires_grad=True, the new total_loss tensor also does. The subsequent computations involving total_loss are then recorded during the entire epoch, and eventually clutter the GPU memory.
The solution is simply to write
total_loss += loss.item()

The type casting of the total_loss variable may have been harmless before the Variable and Tensor classes were merged, but not anymore.
As there is no mean to add in place a tensor to a float, Python falls back to using some __radd__ method of the Tensor class. I think this should at least warn the user about the dangerous path he’s taking.
What’s your point of view on this matter ?