I just skimmed through your code, and stumbled over these lines:
list_predictions += [forecaster_output.cpu().numpy() ]#+ target_tensor[0].numpy()]
# pass list of lists with lists of B predictions
outputs += [list_predictions[0]]

Wouldn’t this just add the first prediction to outputs, while the new ones are appended at the end?
This wouldn’t explain why your output changes, when the optimizer is called, so I probably miss something.