The empty and repeating tensors seem to be from L-BFGS. Adam doesn’t do this.
Using:
torch.backends.cudnn.benchmark = True

Seems to slow down the creation of these “repeating” tensors.
Specifying a history_size, seems to place a hard limit on the creation of these “repeating” tensors.  Is this normal/expected behavior?